The KV Cache Mirage: 1.9x That Frees Nothing

Every team trying to wring the best economics out of a serving stack is reaching for the same lever, and a lot of them are falling for the same trap. It does not matter whether the model lives in a single box (the 8-way B300 nodes reaching general availability right now carry roughly 2.3 TB of HBM) or spans an NCCL fabric across many of them: context and concurrency are capped by the KV cache, so KV-cache optimization is where everyone goes looking for free capacity. When a library advertises more KV tokens at no cost, it lands on the most expensive bottleneck in the deployment. The instinct to take that offer is correct. The discipline to measure it before you build a capacity plan on it is the part this note is about.

The challenge, concretely. One 96 GB RTX PRO 6000 Blackwell, a 120B NVFP4 model, and a goal every operator recognizes: pack more context and more concurrent agents onto silicon that bills by the hour. The candidate was a KV-cache quantizer called TurboQuant, merged into the nightly stack, advertising roughly 1.9x more KV tokens than plain FP8. More KV is more context, or more agents, on the same card, which at fleet prices is a real line item.

How we approached it was deliberately boring. Build a gold reference at full precision, then measure every alternative against it on our own model and our own card, with an acceptance bar set before a single number came in. No coherence spot-checks, no borrowed vendor averages. The result of that discipline was not the one we went looking for, which is usually the sign the discipline was worth keeping. We did not get 1.9x. We got three layers of reality, each a measurement that contradicted the headline, and a quiet winner that was already running.

Plain FP8 keys and values turned out to be the largest budget available, indistinguishable from lossless, and the fastest of the field. There was no trade to make. It is worth walking the three layers, because each one is a trap an operator inherits the moment they take a vendor number on faith.

Coherent Is Not Lossless

TurboQuant ships several presets, and to its credit its own implementation docstring publishes the perplexity cost of each one. Here is what it claims, FP8 as the baseline:

An earlier pass had flagged the aggressive end of this table as the promising win, the 2x-class budget. Read the cost column again. The preset that buys you a bit over 2x KV carries a +10.63% perplexity regression. The next rung up doubles that.

Now the part that makes this dangerous. The output from that preset still reads fine. Coherent sentences, correct grammar, no garbage tokens, nothing a human skimming a sample would flag. That is exactly the trap. A coherence spot-check answers the question “does it still talk?” and that question cannot see a ten percent erosion of quality. You only catch the rot by measuring against a reference, the same model at full precision, on the same prompts.

It gets worse for a hybrid model like this one. The method has a boundary-protection step for hybrid architectures, and that step is disabled by design here. The docstring notes that removing it dropped GSM8K by around 30 points on some models. So the published deltas may understate the real damage on the model we actually wanted to serve.

The chart is the whole point of this section. The KV multiplier you were shopping for and the quality you pay for it move in opposite directions, and the published numbers are the optimistic ones. None of this we measured ourselves yet. This is the vendor’s own homework. We just read it before we believed it.

It Doesn’t Run on This Silicon

So we tried to measure it ourselves anyway, with the correct backend selected. Every preset initialized cleanly, loaded the model, reported its inflated budget, and then died at the first prefill:

torch.ops._vllm_fa2_C.varlen_fwd -> CUDA error:
the provided PTX was compiled with an unsupported toolchain
(cudaErrorUnsupportedPtxVersion)

Same crash, same call site, every preset. The vendored FlashAttention-2 kernel that TurboQuant rides on has no compiled machine code for this architecture, and the embedded PTX fallback is rejected by the driver. This is not a tuning problem. It is a hard wall between what the kernel was compiled for and what we asked it to run on.

Here is the nuance that costs people real time. “Works on Blackwell” almost always means “works on datacenter Blackwell,” the sm_100 parts in the DGX and HGX boxes. The workstation and pro cards, the ones a lot of teams actually have under a desk or in a single-node rack, report sm_120. Different compile target. A large class of kernels, prebuilt wheels, and “supported on Blackwell” claims silently exclude sm_120, and you do not find out until the first forward pass. Our production path, FlashInfer attention, never touches that FA2 kernel, which is exactly why plain FP8 and BF16 serve fine on the same box. The crash is specific to the path TurboQuant requires.

cudaErrorUnsupportedPtxVersion is not a one-off. It is a failure class: something built against one toolchain or architecture, deployed onto another. Today the sharp edge is sm_120 versus sm_100. Tomorrow it is a driver bump, or a CUDA minor-version mismatch baked into a golden image and rolled to a thousand nodes before anyone runs the workload that trips it.

Even If It Ran, It Frees Nothing

This is the layer that retired the idea for good, and it has nothing to do with crashes or quality.

On this model the KV budget is bound by the Mamba state, not the attention KV. That sentence needs the model’s shape to land. Nemotron-3-Super is a hybrid: 40 Mamba2 layers, 40 mixture-of-experts feed-forward layers, and only 8 full-attention layers, those with two GQA key-value heads each. KV-cache quantization touches the attention layers. On this architecture that is 8 layers out of 88, with tiny heads.

So we measured the actual usable budget at a fixed, pinned config: 256k max context, 4 concurrent sequences, 0.95 memory utilization. BF16 keys and values, then FP8 keys and values, everything else identical.

Byte-identical. Not close, identical, down to the token. The allocator log tells you why: it forces the attention page size up to match the Mamba page size, and the float32 Mamba2 state dominates the roughly 12.35 GiB cache pool. Quantizing the keys and values of 8 attention layers with two heads frees no measurable memory, because those bytes were never what filled the pool. KV dtype is not a capacity lever on this model. The only real lever, if you ever need one, is the Mamba state footprint itself, a completely different knob and a separate investigation.

This is adding a bigger fuel tank to a car that is limited by its tires. You can bolt on all the capacity you want. The thing that actually decides how far and how fast you go was never the tank. We went shopping for a 1.9x fuel tank on a car that needed rubber.

(An earlier, looser pass had reported FP8 around 4.17M tokens and TurboQuant around 7.48M. Those came from a different utilization, a different config, and a pre-alignment nightly. Pin one config, measure it honestly, and the gap evaporates: FP8 equals BF16 equals 2.147M, exactly 1.00x. The discipline of fixing the config is what surfaced the truth. The exciting number came from a moving baseline.)

The Incumbent Already Won

Set the failed challenger aside and just grade the default against a lossless reference. BF16 keys and values are the gold baseline, the most faithful the model gets. FP8 is what production already runs.

FP8 is simultaneously the maximum budget achievable, quality-indistinguishable from lossless, and the fastest of the three by a hair. The perplexity delta, +0.54%, sits comfortably under the one-percent acceptance bar we set before running anything. Retrieval is perfect at both precisions, 20 needles out of 20 across contexts to 77k tokens. There was no axis on which the aggressive option could have won, because the default already held all of them.

One credit belongs here, because it is the reason the perplexity numbers above mean anything at all. This model runs its NVFP4 experts on a native four-bit path on the Blackwell tensor cores, not a software emulation that dequantizes the weights back to a wider type and only pretends to be the format. The distinction is not cosmetic. An emulated kernel can drift numerically from the real hardware op, and a perplexity score measured on an emulator is, in part, measuring the emulator rather than the model. Running the format end to end, with activation quantization fused into the kernel, is what keeps cohesion intact and the perplexity honest. That native sm_120 path exists because of Luke Alonso’s b12x, the fused-MoE and FP4 GEMM backend that runs NVFP4 end to end on Blackwell tensor cores (wired into vLLM by PR #40082). It is unglamorous enabling work, and our entire reference baseline rests on it. Worth saying plainly: outstanding job.

A word on how those quality numbers got honest, because the methodology nearly lied to us first. The needle test scored false misses at the start: this model reasons out loud before answering, and a 64-token cap was cutting off the answer mid-thought. We raised the cap to 1024 and matched the code in either the reasoning or the final content, and BF16 went from a suspicious 16/20 to a clean 20/20. The bug was in our measurement, not the model. Separately, the full-length perplexity passage kept running the Mamba2 prefill workspace out of memory at 0.95 utilization and killing the engine, twice, so we capped to a fixed 919-token passage held identical across configs to keep the deltas valid. That out-of-memory is itself a real fragility of running a hybrid this hot, worth one sentence in anyone’s runbook.

What This Costs a Fleet

Four operating lessons survive contact with hardware this specific, and they scale straight up to a fleet.

The free-capacity number is usually measured on someone else’s silicon, with a metric that hides the tail, or against a component that is not your bottleneck. Operators adopt these on faith to stretch expensive cards, and they inherit silent quality regressions that no coherence check will ever surface. The antidote is not skepticism as a personality. It is a gold reference and an on-model delta, every time, before the number goes in a capacity plan. This is the same discipline behind our independent NVFP4 benchmark: the number nobody publishes is the one you measured yourself, on your own hardware, with something at stake in the answer. It is the same reason telemetry that agrees with itself can still be lying.

Compiled-for versus run-on is an inventory discipline, not a footnote. The PTX wall here is one card and one driver. The same failure class is a CUDA-minor mismatch in a golden image rolled across a thousand nodes, discovered by the first production request rather than the build. Architecture, driver, and toolchain are a matched set, and the matching belongs in the inventory, not in tribal memory.

Know where the bytes are before you optimize. We almost spent real engineering on quantizing the wrong tenant of the memory pool. At fleet scale the same mistake is buying the wrong accelerator, tuning the wrong cache, or rightsizing the component that was never the constraint. Capacity planning without a memory map is folklore with a spreadsheet. The lever on this model was the Mamba state, and we would have missed it entirely if the attention-KV story had happened to work.

And the economics, which is where a fleet decision actually lives. Chasing aggressive KV quant to pack more context onto a costly card is a rational instinct. Here it would have cost real quality for zero capacity, and the cheapest, fastest, highest-quality option was the one already running with no change request attached. Predictable beat clever. Sometimes the entire value of an investigation is a defensible “no,” signed, with the measurements behind it, so the next person who reads the same seductive README does not spend the week you already spent.

Keep It Honest

This is one model on one card, and the strongest claim here is the narrowest one. The Mamba-bound conclusion is specific to attention-light hybrids. Put a dense model or an MLA-heavy one on the same bench and the balance swings back toward attention KV, where quantizing it would actually move the budget. Say that out loud, because the wrong generalization from this piece is “KV quantization never helps.” It often does. It just could not help here.

The perplexity table for the presets is TurboQuant’s own published figure, not our measurement. Our measurement is the FP8-versus-BF16 row, and that one is ours end to end. And the PTX wall is true as of this stack and this driver. A future FlashAttention build with real sm_120 machine code would lift it, at which point the first two layers change and the third, the Mamba-bound budget, still stands. That is the load-bearing finding. The other two are just the reasons we never got to argue with it.

The default won on every axis we could measure. The week was not wasted proving that. The week was the proof.