Performance Engineering Is the A in CIA

Every agentic pipeline we have designed this year has the same shape: a chain of serial hops, and somewhere in the chain, a call to a hosted API that does one thing better than anything you could build in a week. The budget conversation is always about the model at the end of the chain. The availability conversation, when it happens at all, is about whether the API is up. Nobody asks what the API costs on a healthy day, per turn, forever, and nobody files the answer under security.

That is the gap this field note is about. The hop in question is the routing decision in a pipeline we designed and benchmarked for a client. The candidate for it was a hosted routing API. In our measurements it was the most accurate option by a clear margin, and it was also the option that would set the floor of every user-visible latency, the one dependency on the critical path we could not fall back from, and the one place where every message would leave the network. We measured it, built two local alternatives on a small GPU, and wired the routing service so the pipeline consumes those instead. The lesson is not “avoid APIs.” The lesson is that performance engineering is availability engineering, and availability is a security property, so the latency budget belongs in the threat model next to the firewall rules.

Federal law already says so, in plainer words than most security programs use. The statutory definition of availability is “ensuring timely and reliable access to and use of information” (44 U.S.C. § 3552(b)(3)(C)). Most programs defend the second adjective and forget the first. A system that returns the right answer a fifth of a second late on every turn is not available in the sense the law wrote down. It is slow, and slow in series is down.

The hop

Every message a user sends into this pipeline has to be routed before it is answered: which specialised model handles this turn, and does the current session holder keep it or hand it over. One decision, about a dozen possible labels, once per message. The decision has to be right, because a wrong route produces a confident answer from the wrong specialist, and it has to be fast, because it sits in series with everything after it. Nothing downstream can start until it returns.

The hosted API did the job well. On a blind exam of several hundred conversation turns it routed 97.4% of them correctly under the grading that matches how the product behaves, and it handled most of the boundary cases from the label descriptions alone. Then we measured it the way a client would see it, one request per turn from a workstation on our network:

None of these numbers is alarming on its own. A fifth of a second is nothing in a web request. The problem is where the fifth of a second sits.

Where the delay adds up

We replayed one ten-turn conversation three times through the hosted API and through a local router, from the same client, back to back. The per-turn numbers were unremarkable. The accumulated line is the one a user feels.

Three replays of one ten-turn conversation, both routers called from the same client. The hosted hop accumulates 5.7 seconds of waiting; the local one, 0.26 seconds.

On the first hundred turns of the blind exam the totals were 19.2 seconds against 0.7. That is a factor of about 28, the same order as the per-turn ratio, and the chart adds no new physics. It only shows what a fifth of a second per turn looks like to the person waiting for the answer.

The second place it adds up is in series with the model that answers. The pipeline’s large model, on a 96 GB card serving eight concurrent requests, reaches its first token at about 225 ms at the median and 336 ms at the 95th percentile. Put the hosted router in front of it and the router-plus-first-token path becomes about 415 ms at the median and at most about 575 ms at the tail, with the router taking 46% of that budget. Put the local router in front and it is 231 and 344 ms, with the router taking 3%. Two caveats, repeated in the methods note: adding two p95s gives an upper bound on the tail of the sum, not the tail itself, and these figures cover the router plus the model’s first token only. The gateway, retrieval and the user’s own network come on top of both bars equally.

Router latency plus the large model’s time to first token, composed from two measurements. The p95 rows are sums of two p95s, an upper bound.

The third place is the one that made this a security conversation rather than a tuning exercise. The hosted design had no fallback, because nobody had built one. If the API degraded, routing degraded, and if routing stopped, the product stopped. We did not own its SLO; we could load it from our side, but we could never learn its ceiling under other tenants’ load. Every message, every turn, would travel to a third party inside a thousand tokens of our own label definitions, which is a data-classification question that the integration had answered by default rather than anyone deciding it. And the integration called a floating “latest” model alias, so the routing behaviour could change under us with no change record on our side.

Availability, confidentiality, integrity. Three letters, one dependency. That is a seam, and seams are where ownership dissolves.

First build: a matmul in memory

The first local router was deliberately the simplest thing that could work. Take a 137-million-parameter sentence embedder, embed a few hundred example sentences written to the shape of chat traffic, label them, and keep the result in memory: a matrix of 768-dimensional vectors, under 2 MB. For each incoming message, embed it, multiply against the matrix, average the three best scores per label, add a small keyword boost, and compare the best label against one threshold. If it clears, that specialist takes the session. If not, whoever holds the session keeps the turn.

No vector database. At a few hundred vectors a flat FAISS index is the same arithmetic plus a dependency, and approximate indexes only pay off somewhere past a hundred thousand vectors. The decision itself is two microseconds. End to end, through the embedding server and the routing service, 3.5 ms at the median on the same workstation that saw 135 ms from the hosted API, in 0.9 GB of GPU memory.

It scored 89.7% on the blind exam. We had bought 92% of the accuracy with under 3% of the latency, and lost 7.7 points doing it.

The lost points had a shape. A nearest-neighbour router measures how much a message looks like each label’s example sentences. It cannot read “and I need one for the office as well” in the light of who is holding the session. The hosted API could, because its input is the current holder together with the message, and its instruction is to stay unless the message clearly belongs elsewhere. The accuracy we lost was exactly the capability we had not built.

Second build: a session classifier on the same interface

So we built that capability, small. A BERT-class encoder of about 150 million parameters, fine-tuned to take the current session holder and the message together and answer one of the labels. The training data was a few thousand conversation turns written to the exam’s shape by LLM writing agents, each briefed on a different user population, none of which saw the exam or the router’s examples; turns overlapping the exam were removed, and synthetic departures were added, where an opener is replayed with a random other specialist holding the session. Training takes under a minute on one card. Selection used a calibration exam; the report exam was graded once. That is the MLOps loop in miniature, generate, train, calibrate, promote, and it is short enough to rerun whenever the label table changes.

The first version scored 93.8% at 4.6 ms p50 through the service. Then we did what performance engineering actually consists of on a model serving path, which is looking at where the 4.6 ms went. The HTTP hop and the Python server cost 0.22 ms. Tokenising, 0.11 ms. The forward pass, 4.1 ms, and eight messages in one forward pass took 10.8 ms, so the single-message time was kernel launch overhead, not arithmetic. Switching the weights to bf16 did nothing on its own. bf16 weights with fixed-length padding and CUDA graphs, which capture the launch sequence once and replay it, brought the forward pass from 4.47 ms to 1.45 ms. The service now answers at 2.1 ms p50 and 4.4 ms p95 from the routing client, and the score moved to 94.1%.

Blind-exam accuracy against median routing latency through the service, log scale. The hosted point was measured from the same workstation against the vendor’s hardware.

Testing found two more things, both of which would have become incidents later. A threaded model server was slower than a single-threaded one, 15 ms against 4.5, because the threads contend for the GPU work and the interpreter; the server stays single-threaded. And the default listen backlog of the model server, a handful of connections that nobody had ever thought about, dropped connections under 32 parallel sessions, producing one-second TCP retransmit tails and spurious fallbacks on a service whose whole point was two milliseconds. A backlog in the hundreds and a 500 ms timeout closed it.

The ceiling is real and it belongs in the article, because the thesis is the tail. The model server answers one request at a time, so parallel sessions queue. Before the bf16 build, 32 parallel sessions saw 141 ms at the median and 372 ms at the 95th percentile, worse at the tail than the hosted API. The current build holds 14 ms at the median at eight parallel sessions and close to 500 turns per second on one small card; we have not yet re-run 32 parallel on it, and batching in the model server is the next step. That backlog, and that queue, are the kind of defect no accuracy benchmark will ever show you. They only exist under concurrency, they only cost you at the tail, and the tail is where availability is decided.

Changing the pipeline to consume it

The replacement is not a drop-in swap. It is a change to what the pipeline is allowed to assume, and three things changed.

The routing service now has two routers behind one API and switches by configuration. The classifier decides every turn. The in-memory matmul loads alongside it as the fallback: when the model call fails or exceeds its timeout, the nearest-neighbour router decides that turn, marks the answer as a fallback, and the session continues. We demonstrated it by killing the model server mid-conversation; the next message was routed by the embeddings and the session carried on. The health endpoint reports the model as down so the fleet knows, but routing does not stop. The limits are worth stating plainly. The model and its fallback share one card and one host, so this covers a crashed process, not a dead host. A hung model still costs the 500 ms timeout on each turn until a circuit breaker is added. And the “down” signal belongs on a readiness probe, not a liveness one, or a load balancer will pull a node that is still routing.

A fair question is why not keep the hosted API and put the matmul behind it. That would have covered an outage. It would have done nothing about the 190 ms floor on every healthy turn, and nothing about the message leaving the network. Fallback fixes the bad day. It does nothing for the normal one.

The latency budget is now owned. Every number above was measured on hardware we control, with a replay tool that runs the exam through the live service and reports median, tail and accuracy together, so a regression can be seen by re-running it rather than discovered in a customer’s session. And the message stays in the building. The data-classification question the integration had answered by default is now answered on purpose.

For the reader who files this under a security domain rather than an engineering one, the mapping is direct. The hosted hop is third-party risk on the critical path, with a vendor SLA standing in for an SLO we never owned. Sending the message out is an egress decision that needs a data classification first, not after. The fallback is a deliberate fail-operational choice, and it carries an integrity cost that has to be written down: the degraded mode routes about four points less accurately than the primary. And the fallback’s value is bounded by failure-domain independence, which today is one process on one card, not two hosts.

The net effect on the one hop: about 30 times less latency at the median on the same client, no per-call bill, no third-party dependency on the critical path, 1.7 GB of GPU memory including the fallback’s embedder. On the router-plus-first-token path, 1.8 times less time at the median and 1.7 times at the tail bound, with the router’s share of that budget down from 46% to 3%. The large model that answers is now the budget, which is where the budget should be.

The asterisk

We did not beat the hosted API on accuracy. It sits at 97.4%; the local classifier sits at 94.1%. The 3.3 points left on the table have a known shape: departures phrased as “and I need one for the office as well” while another specialist holds the session, and tentative mentions that read as a switch but are not. Both are patterns the first round of training data under-produced, and both have a cheap next step. The next engineering step on the latency side is also known: running the encoder inside the routing process through ONNX Runtime or TensorRT, which removes the Python process and lands between 1 and 1.5 ms. We have not built it, because at 2 ms the router stopped being the problem.

So the honest summary is this. A hosted component that is right 97% of the time at 135 ms or more per call was replaced by a local one that is right 94% of the time at about 2 ms from the same client, with a 90% fallback underneath it that costs 0.9 GB of memory and never leaves the machine. For a pipeline whose availability is counted in accumulated seconds and whose confidentiality is counted in what leaves the network, we take that trade, and we would make it again at the next hop.

Methods note

The hosted API was evaluated as a candidate for this hop and benchmarked as the accuracy ceiling; it was never the production router, and the nearest-neighbour router predates the evaluation. The accuracy figures are from one blind exam of several hundred turns, graded once per arm, under stay-by-default grading: a mid-session turn that is not a departure counts as correct when the router kept the current holder. Per-turn grading, which also penalises small talk handed to the general assistant, gives 94.1% for the hosted API and 88.2% for the local classifier. The exams and the training conversations are synthetic: each was written by an LLM agent from the label table and a case list, blind to the router’s examples. They measure routing on traffic-shaped text, not on production logs. The cumulative chart uses per-turn latencies from three replays of one ten-turn conversation, both routers called from the same Python client back to back; through that client the local router measures about 6 ms, against 2.1 ms from the Go replay client used for the service figures. The turn-budget chart adds the router’s p50 and p95 from that client to the large model’s time to first token at eight concurrent requests, measured on a different day on the 96 GB card that serves it; the two were not measured in one run, and the p95 rows are sums of two p95s. Hosted latency was measured from our own network; from a cloud region near the vendor it would be lower, and the median gap would shrink, though not close. The local GPU figures are from a 16 GB workstation card; the CPU-only classifier figure in the chart is the same model run on that workstation’s CPU. The hosted vendor, the client and the product are deliberately not named.