Let me set the terms first. The M5 Ultra is not an inference serving machine, and comparing it with NVIDIA serving as if it were one would be a silly exercise. This piece treats it as what it is: a personal machine, or the box behind a small shop’s handful of agents taking turns. The RTX PRO 6000 is here as the yardstick, because it is the card that tells you honestly what you give up. ...
Every team trying to wring the best economics out of a serving stack is reaching for the same lever, and a lot of them are falling for the same trap. It does not matter whether the model lives in a single box (the 8-way B300 nodes reaching general availability right now carry roughly 2.3 TB of HBM) or spans an NCCL fabric across many of them: context and concurrency are capped by the KV cache, so KV-cache optimization is where everyone goes looking for free capacity. When a library advertises more KV tokens at no cost, it lands on the most expensive bottleneck in the deployment. The instinct to take that offer is correct. The discipline to measure it before you build a capacity plan on it is the part this note is about. ...