Vector Search on CPU, Memory, and cuVS

Everyone reaches for the GPU. At 114 megabytes, the GPU might be the wrong answer.

This is the companion bench to Visual RAG Beats the Vision Model, where a distilled image-retrieval corpus beat a 3-billion-parameter vision model on accuracy at 180 times the speed. That whole argument rests on one quiet assumption: that the retrieval underneath is cheap. Cheap to run, cheap to host, cheap in watts. This article is where I check that assumption instead of asserting it, because an assumption you have not measured is just a hope wearing a lab coat.

The thing being measured

One index. 27,497 vectors, 1,024 dimensions each, L2-normalized DINOv3 features. On disk it is 114 megabytes, which is the entire point. This is not a billion-vector web-scale index where the GPU is obviously the only answer. It is a small, dense, real corpus, the kind most applied teams actually have, and at this size the right hardware is not obvious.

Three ways to search it, same queries, same k:

• CPU, brute force. No index, just the math. Every query scored against all 27,497 vectors. Embarrassingly simple, and at this size maybe embarrassingly sufficient.
• In memory. A proper approximate-nearest-neighbor index held in RAM. The path most vector databases take by default.
• GPU, cuVS. NVIDIA’s cuVS, the GPU-accelerated search library, running CAGRA and IVF indexes on Blackwell. The path everyone assumes you need.

What gets recorded

Not just “which is fastest.” That question is boring and the answer is rigged in the GPU’s favor before you start. The questions that matter on a budget:

• Query latency, p50 and p95, single-query and batched.
• Throughput, queries per second, when you actually push it.
• Build time and memory footprint, because an index you rebuild often pays its build cost again every time.
• Watts. The RTX PRO 2000 in my workstation draws 70 watts. The RTX PRO 6000 in the compute box draws around 300. A CPU answering in two milliseconds at a fraction of the power may quietly beat a GPU answering in two-tenths of a millisecond you were never going to feel.

I will run it on both cards, because the 6x VRAM and 8x memory-bandwidth gap between them is exactly the kind of thing that looks decisive on a spec sheet and turns out not to matter at 114 megabytes.

The hypothesis, stated before the data

I am writing this part before the numbers exist, on purpose, so I cannot quietly rewrite the prediction to match the result later.

My bet: at this corpus size the CPU and the in-memory index are already fast enough for any interactive workload, the GPU wins the raw latency race by a margin nobody will ever perceive, and cuVS only pulls decisively ahead when the corpus grows by one or two orders of magnitude, or the query load goes concurrent and stays that way. If that holds, the lesson is the one the economics keep teaching: match the hardware to the workload you actually have, not to the benchmark you saw in a keynote.

If I am wrong, I will show you exactly how wrong, in the same table.

Why publish before the numbers

Because a benchmark whose prediction can be quietly rewritten after the fact is not a benchmark, it is marketing with a delay. This page is the pre-registration: the corpus, the three backends, the metrics, and the bet are now fixed in public, dated, and the hypothesis above does not get edited once the data lands.

The results will publish here in one table: p50 and p95 latency, throughput, peak memory, and watts, for each of the three backends, on both cards. If the GPU wins at 114 megabytes, that table will say so in exactly the same font as if it loses.

Status (2026-08-11): the bench ran the same day, on both cards. The hypothesis above stays exactly as written; everything below was measured after it was locked.

Numbers when the bench finishes. Not before.

What the numbers said

Disclosures first, because they are the part most benches leave out. The corpus is not the exact cut the protocol described: the bench machine holds an earlier iteration of the index, from before the final distillation pass, 29,193 vectors and 119.6 MB, while the 27,497 and 114 MB written in June describe the distilled set. Same embedding space, same dimensionality, six percent more vectors, and both cards searched the identical matrix. The big card was not empty: an idle model server held 69.9 GB of its VRAM at zero percent utilization and 23 watts, and the bench ran beside it. And the watts column is GPU-only, because the CPU package energy counters on both hosts are root-gated, and an estimated number is worse than a missing one.

One thousand queries drawn from the corpus with a fixed seed, k=10, ground truth exact. Latency is serial single-query over 300 samples; batch throughput is one warmed call over all 1,000. Memory is a non-story at this size: 120 MB as a raw matrix, 127 MB as HNSW, about 400 MB resident on the GPU with graph and workspace.

Scoring the June bet, clause by clause.

“The CPU and the in-memory index are already fast enough for any interactive workload.” True, with margin to spare. HNSW answers in 0.15 to 0.24 milliseconds at 0.997 recall. Even brute force, the no-index option, clears every interactive budget that exists at 1.6 to 3.0 milliseconds, and it is exact. A detail worth keeping: the little Xeon beats the i9 at single-query brute force, 1.57 against 3.0 milliseconds, because a matrix-vector scan is memory-bandwidth-bound and the workstation chip has four DDR5 channels against the desktop’s two. The physics shows up even at 120 megabytes.

“The GPU wins the raw latency race by a margin nobody will perceive.” Wrong, and wrong in the direction I did not concede. The GPU lost the latency race outright. Per-call launch and transfer overhead put CAGRA at 1.6 to 2.6 milliseconds for a single query: seven to seventeen times behind CPU HNSW, and on the workstation, behind brute force. At this corpus size, one query is not enough work to wake the card up for.

“cuVS pulls decisively ahead only when the load goes concurrent and stays that way.” True, and the margin is not subtle. Batched, the RTX PRO 6000 answered 372,000 queries per second, ten times the best CPU number on the bench, while drawing 83 watts on a card rated for far more. The 70-watt workstation card did 54,000 at 40 watts, which works out to under a millijoule per query. When queries arrive as a flood instead of a trickle, the GPU is not incrementally better. It is a different regime.

One methodology confession, in the spirit of the house rule that the bug is usually in the measurement. The first pass recorded the workstation card’s batch throughput as 712 QPS. The true number is 54,000. The first large-batch call pays a one-time workspace allocation, and timing that call as the measurement understated the card by seventy-five times. Three warmed repetitions agreed before the number above went in the table. Had we trusted our own harness on the first pass, this article would have published a GPU seventy-five times worse than it is, and no reader could have known.

So the June bet scored two out of three, and the miss is the useful part: the hypothesis conceded the GPU a symbolic latency win it turned out not to earn. The operating rule survives contact intact. Match the hardware to the arrival pattern of your queries. A single interactive stream, a human or an agent chain waiting on one answer, wants HNSW on the CPU it already has. A sustained concurrent flood, a reranking service, an evaluation sweep, an agent fleet, wants the GPU and it will not be close. And at this size, nobody needs to buy anything: every backend here clears the interactive bar on hardware that was already in the room.

The bench finished. These are the numbers.