M5 Ultra vs RTX PRO 6000 for LLMs: Measured, Priced

Let me set the terms first. The M5 Ultra is not an inference serving machine, and comparing it with NVIDIA serving as if it were one would be a silly exercise. This piece treats it as what it is: a personal machine, or the box behind a small shop’s handful of agents taking turns. The RTX PRO 6000 is here as the yardstick, because it is the card that tells you honestly what you give up.

The question that does deserve an answer comes from anyone running models for themselves: why buy an accelerator card when a Mac Studio with 256 GB of unified memory holds the whole model on a desk? Unified memory quietly removed the capacity argument for a whole class of mixture-of-experts models. The usual answer is a shrug and “the Mac is slower.” That is true. It is also the least useful thing anyone can say about it, because it tells you nothing about which work cares.

So we put the same model on both machines and measured where the gap actually lives. The model is Qwen3.6-35B-A3B: a 35-billion-parameter MoE with roughly 3 billion parameters active per token, built from hybrid Gated-DeltaNet and attention layers. On one side, an RTX PRO 6000 Blackwell Max-Q in a workstation, serving NVFP4 through vLLM, already tuned for production. On the other, a Mac Studio M5 Ultra with 256 GB of unified memory, serving through MLX and through vLLM on Metal. Same real workload, same client, same metrics, same evening.

The finding is not that the Mac is slower. It is where, and why. Measured like for like, the Blackwell card runs a decode step 1.7 to 2.8 times faster depending on batch size, and prefills about three times faster. End to end, stock software on the Mac delivered about half the card’s throughput for a single user; a patch we wrote to mlx-lm raised that to 78%. Under load it stays near 30%, patch or no patch. Memory bandwidth explains none of it, on either machine. And at the chip, per joule, the Mac wins outright for a single user and lands within about 15% of the card under load, because the card spends most of the evening pinned against its 300 W cap.

I had a different headline for this piece a few hours earlier, and the section on decode explains why it did not survive.

Why an NVIDIA Guy Bought a Mac

I should declare the bias. I have been an NVIDIA fan for a long time. I never stepped into in-memory ONNX and that world. Even for small matmuls I was reaching for cuVS on Blackwell, mostly for research, chasing the fastest RAG I can build in the petabyte range. I will still get there.

But anyone who has spent time with token generation knows how much of the conversation comes back to memory bandwidth, and decode most of all. So when Apple launched a machine with 1.2 TB/s of memory bandwidth, I was really surprised. I ordered an M5 Ultra on the first day orders opened, and it took two months to arrive. For the first time I stepped out of corporate serving and into something else: local inference I can call mine. I still feel orphaned from some CUDA and TensorRT things I use in my daily projects, nothing an SSH session doesn’t solve, and it has been a very interesting walk.

The number that sold me is not the number that decided this comparison. We measured the box at about 1.1 TB/s, close to the spec sheet, and then watched single-stream decode use a quarter of it. This model is a sparse MoE with about 3 billion parameters active per token, and a dense model should lean harder on the memory, but that is not what we measured here. I bought the Mac for its bandwidth. The results below are about everything else.

A Real Workload, On Purpose

A fixed 512-in, 128-out benchmark tells you about kernels. It tells you nothing about a shopping assistant whose tenth turn drags the nine before it.

We replayed real multi-turn shopping-assistant conversations across four product niches, ten turns each. Every turn carries a retrieved fact sheet of about 1,400 tokens plus the conversation so far, so prompts average about 11,300 tokens and the longest reaches about 23,000. Temperature 0, a 200-token output cap, thinking off, streaming on. That shape matters: it is prefill-heavy, it rewards prefix caching, and it looks a lot more like production retrieval traffic than a chat demo does.

The metrics are the ones a serving SLO is written in: time to first token (TTFT) at p50 and p95, time per output token after the first (TPOT), aggregate output tokens per second, and prefix-cache hit rate. Concurrency levels 1, 8, 16 and 32, with 24 conversations per level (32 and 64 at the two highest levels, so every in-flight slot sees at least two conversations). Each level starts from an empty prefix cache. On the Mac, 16 and 32 are stress levels to find the edges, not a use case anyone should plan around. The client is the same small, standard-library-only harness on both machines, byte for byte, replaying the same prompt file.

Two probes sit alongside the workload, again one script run against both vLLM servers. Pure prefill: a fresh prompt of 2k, 8k or 16k tokens, one at a time, with a random prefix so nothing hits the cache, and a single output token. Pure batched decode: 1 to 64 short prompts at once, 512 forced output tokens each, timed over the middle of the generation when every stream is decoding and nobody is prefilling.

Decode Is Not Where the Mac Keeps Up

Start with a single stream, where the hardware has nowhere to hide.

The Blackwell card, running the native b12x FP4 MoE kernel, does about 261 tokens per second single stream, or 3.8 ms per token. At concurrency one the card is ahead by roughly 1.7x on decode. Nobody should be surprised.

What should surprise you is why. We measured the Mac’s memory bandwidth at about 1.1 TB/s (1,132 GB/s on a read, 1,074 GB/s on a copy, 4 GiB buffer, our measurement rather than a spec sheet). At 157 tokens per second, single-stream decode moves roughly 270 GB/s of active 4-bit weights. That is 24% of what the memory can do. The 8-bit build reads 1.8 times the bytes and is only 1.3 times slower. Single-stream decode on the Mac is not waiting on memory. It is waiting on the fixed cost of launching each step. The Blackwell card is in the same position for the same reason: 261 tokens per second moves about 440 GB/s, roughly a quarter of its rated 1.8 TB/s.

A fixed per-step cost is exactly what batching amortizes, so the obvious next question was how each machine’s step grows as you add sequences. An earlier version of this draft answered it badly. It set the Mac’s pure decode step against the Blackwell card’s per-token latency under the real workload, found them level from eight sequences up, and called it parity. The Blackwell number in that comparison was already paying for prefill sharing the card. That is not a comparison; it is a handicap. So we took the card out of production for an evening and measured its pure decode step the same way.

Figure 1. Pure decode, no prefill on either side. The gap is smallest for one user and widens as the batch grows.

There is no parity anywhere in that table. Both machines batch well, which is the overhead story from above playing out: going from one sequence to 32 costs the card three times the step time for 32 times the tokens, and costs the Mac four and a half times. But the card’s step stays shorter at every size, and the gap grows with the batch, from 1.7x at one sequence to 2.8x at 64 against the Mac’s best kernel. The Mac’s step grows faster because each extra sequence adds real arithmetic (expert matmuls, attention over its own cache), and that arithmetic is exactly what the next section is about.

Note the third column, too. vLLM’s own decode step on Metal is slower than the bare MLX kernel underneath it at every batch size, by 2.5x at one sequence and about 20% at 32. That is serving-stack overhead, and it comes back later.

Prefill Is Where Physics Shows Up

The same probe, prefill only: one fresh prompt at a time, nothing cached, first token as the stopwatch.

Three to four times faster than vLLM on Metal, and two and a half to three times faster than the bare MLX kernel. The reason is arithmetic, and it is hardware-deep. Blackwell runs FP4 math on its tensor cores with the activations quantized too, so prefill, which is a big dense matrix multiply over the whole prompt, executes in the packed format. Apple’s GPU has no FP4 math. On the Mac, 4-bit is a storage format, and every multiply happens in a wider type after the weights are expanded.

Think of 4-bit weights as vacuum-packed clothes in a suitcase. On Blackwell, the tensor cores can wear them straight out of the bag. On the Mac, every garment has to be unpacked to a wider type before anyone can wear it. The packing still pays off (it is a big part of why 256 GB holds this model so comfortably), but the unpacking is charged on every use. During single-stream decode, when each step needs a handful of shirts and the walk to the closet dominates, you barely notice. During prefill, when you are dressing ten thousand tokens at once, the unpacking is the job.

Now the end-to-end numbers, same workload, same client, same evening.

* mlx_lm reports cached tokens per request; vLLM reports block-level hits. Close, not identical.

Pick the best Mac engine at each level and the scoreboard is short:

Those percentages barely moved from the smaller screening runs we did first (51, 31, 31 and 30). The shape is stable, and it is not subtle.

At eight concurrent conversations the Mac’s pure decode step under vLLM was 18.2 ms. Its TPOT on the real workload was 43.1 ms. The difference is prefill: new turns arriving with a couple of thousand uncached tokens each, and the GPU stopping to chew through them while every in-flight conversation waits for its next token. The card pays the same tax, 6.3 ms pure against 12.8 ms under load, but it pays it in a currency it has three to four times more of. TTFT tells the same story more directly: about three times the Blackwell figure for vLLM on Metal at every level. mlx_lm.server gets closer for a single user, 1.9 times, and falls much further behind under load.

At 32 concurrent the Mac’s serving engine started to fray. In this run vLLM on Metal lost 10 of its 64 conversations on their very first turn, each an empty response when 32 brand-new conversations arrived at once; the smaller screening run at the same level had served all 640 requests without an error. mlx_lm.server did worse in the screening run and was not repeated: its generation thread died on a Metal out-of-memory error and 61 of 255 requests failed, on a machine with 256 GB of memory serving a 20 GB model. That is allocator behavior, not real exhaustion, and it is the cleanest illustration in the whole set of the difference between a single-user server and a serving engine. The first one is very good at what it was built for. Then you ask it for something else.

Which leads to the result that surprised me most. On the Mac, the NVFP4 build of the same model decodes slower than plain affine 4-bit: 142 tokens per second against 157 in the first run, 151 against 163 in a later clean session. Seven to ten percent, depending on the day. On the Blackwell card, the same evening, NVFP4 beat two INT4 builds of the model on the same card: 3.9 ms per token single stream against 4.5 and 5.0, and about 26.6k tokens per second of prefill against 22.4k, with batched decode a tie against the better of the two. NVFP4’s advantage comes from hardware that consumes the format natively. Without that hardware, it is just a more elaborate way to unpack. A quantization format is not a property of the model. It is a contract with a specific piece of silicon, and the NVFP4 benchmark we ran earlier on Blackwell only holds on the side of the contract that has the tensor cores.

Serving Software Decided More Than Silicon

At low concurrency, the biggest single swing in this whole exercise came from a configuration flag, not a chip.

vLLM’s Metal plugin, out of the box, showed 40.2 ms per output token at concurrency one. That number made no sense against a 6.6 ms decode kernel. The cause: greedy sampling steps were falling back to a synchronous PyTorch sampler, so every token paid a round trip the kernels never needed. Setting VLLM_METAL_NATIVE_SAMPLING=1 took it to 10.9 ms in that session. An adjacent compiled-MLP option, which sounds like it should help, made things worse at 13.5 ms. Both are opt-in. Neither announces itself.

Even with the flag on, this engine’s single-user number would not sit still. Across sessions the same configuration measured 10.9, 14.3 and 14.9 ms at concurrency one, while mlx_lm.server on the same machine reproduced its 7.8 to 8.0 ms every time. The plugin’s per-step work runs partly on the CPU, so anything else the machine is doing leaks into it; on the evening of the final run a browser was playing video on the same Mac. At one sequence that CPU-side overhead is most of the step, which is why the plugin’s pure decode step sits at 16.8 ms over a 6.6 ms kernel.

mlx_lm.server had its own version of the same story. Its default prompt cache holds 10 sequences. With 8 concurrent ten-turn conversations, each one carrying its own growing prefix, ten slots thrash constantly, and the prefix-cache hit rate collapsed to 29%. Raising the cache to 64 sequences brought it back to 83%, cut TTFT from 2.5 s to 0.95 s, and halved TPOT from 69 ms to 33 ms. One integer. Nobody chose 10 badly; it is a sensible default for a single user chatting, which is exactly the workload it was designed around, and exactly not ours.

We also tried shrinking mlx_lm’s prefill chunk size to protect decode latency, and it did nothing, which took longer than it should have to understand. Its scheduler runs one decode step, then one prefill chunk, then one decode step, strictly in sequence, so it behaves like a one-lane bridge with alternating traffic: slice the trucks thinner and the cars still wait for every truck, and everyone’s prefill time lands in everyone’s TPOT no matter how you cut it. vLLM fuses decode and prefill into a single forward pass under a token budget, which is why it degrades more gracefully as concurrency climbs. On Metal the fused steps carry a large fixed cost of their own: shrinking the token budget to 2,048 made TPOT worse (55 ms against 42), and async scheduling changed nothing. That fixed cost is the next thing to profile.

The CUDA side was not immune, and that matters for the argument. When we tuned the Blackwell card for production, the model card’s own launch command silently fell back to a weight-only Marlin kernel and crashed at 16 concurrent requests. The native FP4 MoE kernel, b12x, ships in the stock wheel but has to be requested explicitly. BF16 KV cache beat FP8 on prefix hit rate, because the hybrid cache block came out smaller. MTP speculative decoding made it slower, not faster. Different platform, same lesson: the startup log is the only honest description of which kernels and code paths actually engaged.

Two Mac-only gotchas for anyone’s runbook. After a network change, vLLM’s Gloo rendezvous retried for 8 to 9 minutes on startup before serving anything; pinning it to loopback with GLOO_SOCKET_IFNAME=lo0 VLLM_HOST_IP=127.0.0.1 fixes it. And macOS’s default shell, zsh, starts every background job (command &) at nice +5. A server launched that way, or launched by any tool that was itself started that way, ran at 30 ms per token instead of 15, most likely because macOS steers niced work toward the efficiency cores. Launch long-running servers from bash or a foreground terminal, and check ps -o nice before you trust a number.

None of these are bugs in the usual sense. They are defaults written for someone else’s workload, and the system around them gave no signal that the fast path was off.

Speculative Decoding Flips Sign

On the Blackwell card, MTP made things worse. On the Mac it is the single best decode result we have.

MTP uses the model’s own multi-token-prediction head as a drafter: guess a few tokens ahead, verify them in one pass, keep the ones that match. Single stream, mlx-vlm, a 2,000-token shopper prompt with a long answer:

Same technique, opposite sign, and the reason is the one we already found. The Blackwell card’s single-stream step is already short, so speculation mostly added work there, and it cost 12 to 19 points of prefix hit rate on top. The Mac’s single-stream step is overhead-bound, and verifying four tokens in one pass spreads that overhead across more output. Past a block of four, acceptance drops faster than the amortization pays. At 5.2 ms per token against the card’s 3.8, MTP cuts the Mac’s single-stream decode gap nearly in half.

The catch is getting it together with prefix caching. mlx-vlm can do both, but its prefix cache is off by default, and once switched on it keeps only two recurrent-state checkpoints unless told otherwise, so eight concurrent conversations evict each other down to zero hits. Configured (64 checkpoints, one every 1,024 tokens), it held a 79% cache hit rate at one user with MTP on, and MTP took its TPOT from 11.3 ms to 7.4. That is a real 1.5x. It is also no faster than mlx_lm.server without any speculation (7.8 ms, and a 333 ms TTFT against mlx-vlm’s 575), because mlx-vlm’s own base path is that much slower, and at eight concurrent its batched speculative path stalled prefill badly enough to push TTFT past 15 seconds. vLLM’s Metal plugin only offers n-gram speculation for these hybrids, and n-gram was 2x slower at concurrency one and 4x slower at eight, with 43% acceptance and partial accepts re-scanning the whole batch. MTP on top of the fast serving path, with the cache intact, was the biggest open win on the Mac, so we built it.

We put the MTP head into mlx_lm.server and solved the one hard problem a hybrid model poses. When a draft is only partly accepted, the recurrent layers have to rewind to the state after the last accepted token. Attention caches can simply be trimmed; recurrent state cannot. We made the recurrent kernel record its state after every step of the verify pass, using the same arithmetic as the normal kernel, so a rewind is a lookup rather than a recompute. The prefix cache is saved at the boundary each new turn re-renders identically, and it carries the drafter’s state with it. On the same 24-conversation workload at one user:

Decode is 1.7x faster and the cache hit rate is unchanged. We first wrote that greedy output was token-identical to decoding without speculation, because it was on our four test prompts. Then we ran a new synthetic prompt set, and all four diverged within 200 tokens, each time at a near-tie where the top two tokens sat within 0.125 logits. Stock mlx-lm does the same to itself: splitting the prefill differently changed its output in 5 of 6 conversations, against 2 of 6 for MTP. So the honest claim is that speculation adds no divergence beyond what batch shape already causes, not that the bytes match. That puts the Mac’s single-user decode at 4.5 ms against the card’s 3.9. The cost was concurrency: this first version handled one request at a time, so from about four users up, plain batching served more total tokens.

So we added a scheduler. An idle server decodes with MTP. Once two requests are waiting, the stream in flight moves to mlx-lm’s batcher in the middle of its generation, carrying its cache so nothing is recomputed, and the waiting requests join it. When the batch drains, the next request gets MTP again. The first token after the seam comes from the batcher, and it can differ from what MTP alone would have written, the same near-tie we see from batching shape on stock mlx-lm. Same workload, 24 conversations, a fresh server per level, each cell TTFT p50 / TPOT p50 / output tokens per second:

At one user that is 78% of the card’s 196 tokens per second, up from 51%, and that is the number that changed how I think about this box. At two users TTFT looks worse than stock, but each request finishes sooner, 1.97 s against 2.48 s, which is the number an agent waits on. At eight there is no real win: 186 against 179 in throughput, 6.49 s against 6.21 s end to end. Speculation shortens decode. It does nothing for prefill, and under load prefill is the bill.

One later change had nothing to do with speculation and paid more than any of it, on a different workload: a job queue where groups of eight jobs share one fact sheet and answers run 16 tokens. mlx-lm’s batcher pads every prompt in a prefill step to the longest one, so a single uncached 2,000-token prompt made seven 16-token cache hits cost 2,000 tokens each. Separating long and short prompts into different steps, with concurrent requests waiting on a shared checkpoint instead of prefilling it twice, took eight workers from 14,500 to 39,196 jobs per hour, and to 43,108 once the prompt was laid out with explicit cache checkpoints.

None of this makes us first. A native MTP pull request has been open against mlx-lm since March, others have proposed the same exact rollback, the maintainer has a trimmable cache in draft, and vLLM’s Metal plugin has merged its own. It is also a runtime patch pinned to mlx-lm 0.32.2 and to Qwen 3.5 and 3.6, not a product. The code is public at sch0tten/mlx-mtp for anyone who wants to check the method or take the handoff and the padding fix further.

Prefix Caching Behaved Identically

On hybrid models, vLLM aligns its prefix-cache blocks to the recurrent state, which for this model works out to 1,056-token blocks. Both machines landed at a 77 to 79% hit rate, level for level. Same model, same prompt shape, same block size, same answer. That is the block-quantized ceiling for this workload, not a property of either platform, and it is the same Mamba-alignment mechanic that decided the KV budget in our cache quantization work. If you want more cache reuse here, the lever is prompt layout, not hardware.

Per Joule, the Picture Changes

Speed is one axis. We also logged power on both machines, once a second, for every level of the workload: board power from the NVIDIA driver on the card, and the SoC’s own CPU, GPU and Neural Engine power on the Mac. Divide output tokens per second by watts and you get output tokens per joule.

Two things stand out. The Mac is more than twice as efficient for a single user, and still ahead at eight. From sixteen up the card pulls ahead again, by 12 to 16%, because batching is where it turns its extra arithmetic into tokens. And the card sits at 299 W from eight concurrent upward, which is its power limit: the Max-Q edition is capped at 300 W, and under this workload it lives there. That is the bottleneck nobody puts on the spec sheet. Under load the card is not waiting on memory or on kernels. It is waiting on its own power budget, while the Mac, drawing under 100 W, is waiting on arithmetic and on its serving stack.

These are chip-level numbers, not wall power, and the meters are not the same instrument. The card’s figure includes its own memory and power delivery; the Mac’s covers the SoC compute blocks but not its DRAM. Neither includes the rest of the box: the workstation’s CPU package alone idled at 55 W through the whole run. Wall power would move the comparison further toward the Mac, not away from it, but that is a measurement we have not made, and cost per token needs it plus real prices. Read the table as “comparable at the chip,” not as an electricity bill.

The Room Is Part of the System

Per joule stopped being an academic number once people started doing this at home. Stack up enough power-hungry cards, daisy-chain them off whatever circuit is nearby, and weird things start to come up. Sometimes that includes weird smells.

The arithmetic is not exotic. A common US household circuit is 120 V at 15 A, or 1,800 W, and the usual rule for continuous load is 80% of that: 1,440 W. Four RTX 3090s at their stock 350 W are 1,400 W before you count a CPU, fans or a power supply’s own losses. That build is already at the limit of the wall plate, for hours at a time, on wiring that was run with a vacuum cleaner in mind. Inference is not a burst load. A busy model sits at the cap, as our card did at 299 W from eight concurrent upward.

Then there is thermal soak, which is the part that catches people, because it does not show up in the first benchmark. Every watt that goes in comes out as heat, into a room that usually has no plan for it. The card reads fine for the first ten minutes. Then the air around it warms, the case warms, the intake warms, and the same load runs hotter an hour later than it did at the start. Our 300 W Max-Q, one card in a proper workstation chassis, reached 85 °C during a bench run. We did not log room temperature, wall power or the Mac’s temperatures, so I will not put numbers on the comparison. But a machine drawing under 100 W at the chip is a different conversation with your room, your circuit and the person who shares the house with you.

None of this is an argument against cards. It is the same physics that runs a data hall, arriving in a spare bedroom without the cooling plant, the breakers or the people who watch them. Physics doesn’t negotiate at home either.

A Note on Today’s Prices

Added on November 5, 2026, because this part of the argument moved under our feet while we were measuring. One flag first: everything here is lab and research hardware, bought to learn and to measure. None of this is a production procurement recommendation. A production buy would also price support contracts, redundancy, remote management and ECC across the whole memory path, and the Mac does not offer you all of those.

The market, as of this week:

The memory for a server now costs about what the Mac costs whole, CPU, GPU, power supply and box included. A year ago nobody would have written that sentence, me least of all.

Why the Comparable Box Gets 256 GB

A 96 GB card sounds like it should not care about system RAM. Then the new generation of models arrived. Qwen3.8-Flash-Next carries a 51B n-gram embedding table beside its 125B main model, and the table is meant to live in host memory: 47.7 GiB in FP8, at least 64 GB free for SGLang on this card, and about 102 GB if you keep it at its original BF16.

So 256 GB is not the table’s minimum, and I want to be plain about that. One builder runs this model on a single card with a 64 GB host, by shrinking the table to INT4 or backing it with NVMe. Our own run fit on 128 GB, with 50 GiB left once the table was pinned, after we had shut down three virtual machines and shrunk two more to make room. What 128 GB does not give you is margin: another builder’s kernel ran out of memory on a 123 GB host with two pinned tables, and it took a power cycle to recover. The next sane size for a box meant to hold models like this, with a page cache for 126 GiB of checkpoint, is 256 GB. It also happens to be the number on the Mac, which is the point of pricing them side by side.

What a Comparable Box Likely Costs

If you were starting today, this is the likely bill for a workstation to match the Mac: same memory, about the same core count, same disk in capacity and in speed. For the disk we measured the Mac rather than trusting a spec sheet: 12.8 GB/s sequential read and 12.7 GB/s write (fio, 1 MiB blocks, queue depth 16) and 190k random 4K read IOPS at queue depth 32, on a 2 TB drive 73% full. With a single request in flight it reads at about 8 GB/s. A PCIe 4.0 drive cannot reach that, so the build needs PCIe 5.0.

Against that, a Mac Studio M5 Ultra with 256 GB likely comes to $9,500 to $9,900 depending on storage. We use $9,900 below, which counts against the Mac. That is 3.2 times the money. The build is also a different class of machine, and it should get credit for that: 96 GB of VRAM on top of the 256 GB, ECC, open PCIe slots, four free memory slots, and real upgrade paths. Nothing inside the Mac can be changed after you buy it.

Now divide the throughput from earlier by the price of the whole box:

I did not expect that table. Priced as whole machines, the Mac returns 2.4 times the throughput per dollar for one user and stays slightly ahead at eight agents taking turns, where it is already near the top of what I would ask of it. For an avid user or a small shop, it has quietly become the best bucket for the money.

That ratio says nothing about serving, and I don’t want it read that way. The build does three and a half times the work in one box, at a third of the TTFT, and a serving SLO is written in latency, not in tokens per dollar. If your traffic needs that, buy the card: three Macs do not give you one fast queue. And these are spot prices in a shortage. Check them again before you sign anything.

Ninety-Six Gigabytes Got in the Way

There is a second reason this box changed how I work, and it has nothing to do with tokens per dollar.

For a good stretch, 96 GB was exactly the right size. The runners-up of the local and on-premises playground fit it natively: about 70B at FP8, about 120B at FP4. Llama 3.3 70B at Q8 is 75 GB and gpt-oss 120B at MXFP4 is 63 GB. One card, whole model, room left for KV. Those models are no longer the competition. The new small runners, the ones chasing frontier-like capability at a size one owner can still hold, landed just above the line:

Flash-Next is the honest edge case, because it does run on one card, and several builders now do it well. In the qualified shape we ran, with the table in host memory, it took 96% of the card’s memory and left 3.8 GB free, with four concurrent requests configured against a KV pool that holds about one and a half full contexts. It decoded at a median of 157 tokens per second, which is fast, and others report much more. It was also a machine with no slack left in it, and that is a poor place to do research.

Then we ran the same model on the Mac, with the same 12,752-token prompt. The 4-bit build loaded with the whole table on two stock engines and left 95 to 120 GB of the GPU budget free. With its own draft head it decoded at 74 tokens per second, against the card’s 170 on the same prompt at the same context length, and it was ready in about 13 seconds against the card’s 17 minutes. That is the whole trade in one sentence: the card is faster at what it can barely hold, and the Mac is slower and has room.

DeepSeek-V4-Flash is the clean case. The 4-bit build loaded on the Mac in 15 seconds, with no special engine, and decoded at about 48 tokens per second behind an 11,900-token prompt. That is one untuned session, not a benchmark, and 48 is not 157; they are different models, and the card would be faster here too if the model fit. It doesn’t. The next step on the NVIDIA side is a second card, which at today’s prices takes the build above to about $47,000.

I will say the personal part plainly. Working in LLM inference with only one RTX PRO 6000 has become frustrating, and I did not expect to write that about a 96 GB card. What fits on it natively is no longer the cutting edge of open weights. Nothing got worse: it is still the faster machine at everything it can hold. The models moved, and the card now sits in the middle of the road, too small for the new class and more than the old class needs.

The Mac has a ceiling of its own, and it isn’t far away. The reported 1.6T DeepSeek V4-Pro and the reported 2.8T Kimi K3 run to hundreds of gigabytes even at 4-bit, past this machine as well. The window where 256 GB of unified memory wins on capacity is roughly the class from about 125B to about 300B, and I would not bet on that class staying where it is. Today it happens to be where a lot of the interesting work is.

Where Each Machine Belongs

Actually, one operational number belongs ahead of the placement argument, because it changes how the machines feel to run. Cold start on the Mac is about 12 seconds. On the CUDA box it was about five minutes with a cold compile cache. That gap matters for scale-to-zero, for failover, and for how fast a team can iterate on a serving config. A five-minute restart turns every configuration experiment into a scheduled event.

If you are serving users, this section is short: buy the card. The Mac is not in that race. For everything else, the decision stops being “fast versus slow” and becomes a question of workload shape:

• Prefill-heavy, concurrent, TTFT under an SLO. Long retrieval contexts, multi-turn agents, many users at once. This is where FP4 tensor-core math earns its keep, and the Blackwell card is the right tool by a wide margin: three and a half times the throughput at 16 and 32 concurrent, a third of the TTFT, and better tokens per joule once the batch is big enough to use it.

• Decode-heavy, low concurrency, short new context per turn. Interactive single-user work, evaluation runs, development loops, a small shop’s few agents taking turns. This is the job the Mac is for. Here the Mac delivers half the card’s throughput for one user at under a quarter of the chip power, or 78% of it with the MTP patch, its cold start is a gift, and the memory headroom is absurd: the vLLM KV pool on the Mac held 7.8 million tokens, with room left for models that would not fit on one card at all.

• Models above 96 GB, one owner, research pace. The new small runners in the 125B to 300B class. Here the Mac simply holds the model, and a single card either cannot or only just can.

Either purchase is defensible when it is matched to the workload it will actually serve. The mistake is buying on the single-stream headline in either direction: dismissing the Mac because it decodes slower at concurrency one, or treating it as a drop-in replacement when the traffic is ten-thousand-token prompts arriving eight at a time. At that shape, fully tuned, it delivers about 30% of the card.

Keep It Honest

The caveats, all of them, because the comparison is only useful if its edges are visible.

The vLLM versions differ: 0.27.1 on CUDA, 0.32 plus the Metal plugin on the Mac. The Blackwell numbers are a whole-card bench taken the same evening as the Mac’s. In production that instance runs at 59% of VRAM and shares the card with OCR and speech models; its post-cutover smoke test at 8 concurrent gave TTFT of 225 / 336 ms and 12.9 ms TPOT, close to the bench but not identical. The card is the Max-Q edition, power-capped at 300 W with a blower cooler, and the workstation’s system RAM runs at 4,400 MT/s, so this is not the fastest Blackwell configuration you could build; a 600 W card would likely move every throughput number above sixteen concurrent. The checkpoints differ too: NVIDIA’s official NVFP4 build, with FP4 weights and activations on the experts, against community MLX conversions of the same base model. Greedy decoding is not bit-reproducible under batching on either stack, and output quality was spot-checked, not graded.

On the Mac side: the vLLM Metal plugin’s single-user numbers varied by about 40% between sessions, and the final run shared the machine with a browser playing video; the mlx_lm rows, which are the ones the single-user conclusions rest on, reproduced their throughput and TPOT within 3%. The c=32 rows for mlx_lm.server come from the earlier, smaller screening run. The DeepSeek-V4-Flash run was a single session on mlx-vlm, untuned, one request at a time; the Flash-Next runs were a community SGLang fork on the card and 4-bit builds on mlx-vlm and llama.cpp on the Mac, one request at a time, at temperature 1.0, with different precisions on each side. We did not grade output quality for either, so “frontier-like” here is the field’s description, not our measurement. Power figures are chip-level from two different meters, as described above, and cover stock mlx_lm and vLLM on Metal, not the MTP patch. The hybrid sweeps ran one to two days after the Blackwell numbers, with the same client and prompt file. The price tables mix launch configurator, list, retail tracker and undated listing figures. The $16,000 is the family’s list price: sources do not track the Max-Q separately, and mid-2026 retail listings we saw for it ran $11,000 to $15,000, which would bring the build to about $26,000 to $30,000 and not change the shape at one user. The throughput in the per-dollar table was measured on our Max-Q in a 6-core Xeon workstation with 128 GB, not in the Threadripper build priced beside it. Probes ran through each machine’s vLLM server, so they include serving overhead on both sides; the Mac’s bare-kernel numbers are shown next to them for that reason.

None of that moves the shape of the finding. Where the Mac falls behind, it isn’t bandwidth. It is arithmetic its GPU was never built to do, per-step overhead in a young serving stack, and, until we read the logs, defaults built for somebody else. For one person or a few agents, those are costs you can live with. For serving, they are why you buy the card.

Silicon sets the ceiling. Serving software decides how close you get to it.