Applied AI Engineering for Infrastructure

Everyone reaches for the GPU. At 114 megabytes, the GPU might be the wrong answer. This is the companion bench to Visual RAG Beats the Vision Model, where a distilled image-retrieval corpus beat a 3-billion-parameter vision model on accuracy at 180 times the speed. That whole argument rests on one quiet assumption: that the retrieval underneath is cheap. Cheap to run, cheap to host, cheap in watts. This article is where I check that assumption instead of asserting it, because an assumption you have not measured is just a hope wearing a lab coat. ...

A three-billion-parameter vision model looked at a reCAPTCHA tile and got it right 89 percent of the time. It took 128 milliseconds. A lookup over a few hundred megabytes got it right 95 percent of the time. It took seven-tenths of a millisecond. Same tiles. Same held-out set. One of those is how almost everyone is wiring computer vision into their stack this year. The other is how you should. ...

Since 2023, I’ve been studying applied AI almost exclusively. I don’t pretend to be a data scientist or ML engineer. Honestly, I don’t think giving up more than twenty years of infrastructure, performance, and security engineering would be smart. I’d end up like a duck: swims, flies, and walks, but doesn’t outperform at any of them. It’s impossible not to get caught up in the vibe-coding thing. I’m not here to criticize anyone shipping and prototyping. A few months back, I heard one of the smartest things anyone’s said about AI, from Naval Ravikant. I’ve been listening to him for a few years now, and his takes are consistently good. I don’t remember the exact words, and I’m not going to chase videos or quotes to nail them down, but it was close to this: “There is no disruption caused by AI. The novelty we’re seeing is the abstraction and conversion of human language into computing language.” Brilliant. ...