The second before the first token

12 November 2026 · Bristol

Last spring I watched a team spend four months on answer quality. They built a 900-example eval set, moved from a mid-size model to a large one, gained 4.2 points, and shipped. Internal usage went down.

The reason was visible in the first session recording I looked at. You type a question, press Enter, and then nothing happens for about two and a half seconds. At two seconds people start wondering if it worked. At three they press the button again, which queues a second request and makes the first one slower.

Five numbers, not one

What gets called "latency" is at least five separate things, and only two of them are the model's fault.

Time to first token is the one users feel. Under about 700 ms, an interface feels immediate. Past that, people notice, and past 1.5 s they start editing their question while they wait, which usually makes it worse.

Inter-token latency decides whether the answer reads or stutters. Reading speed is roughly 5 tokens a second. If generation beats that, nobody waits mid-answer and the rest of the budget stops mattering for that turn.

Prefill scales with what you send, not what you get back. This is where retrieval quietly ruins everything. I have seen a 400 ms request turn into 4.1 s because somebody decided to be safe and stuff twelve documents into the prompt. Quality went up by one point. Nobody waited.

Queue time is the one that never appears in a benchmark. Under load, requests wait before anything starts, so a dashboard showing 300 ms of inference can coexist with users seeing three seconds.

Network and render add the last hundred milliseconds, and get blamed for the first two thousand.

What I actually instrument now

Every request logs the five numbers separately, plus prompt tokens and output tokens. I report p50 and p95, never the mean, because the mean is the one number that cannot tell you about the request that took nine seconds.

Two rules came out of this and I apply them before choosing a model:

Set the budget first. If the ceiling is 800 ms to first token, most large models on most hardware are already excluded, and it is much cheaper to learn that on day one.

Measure with production-shaped prompts. Benchmarks use short inputs and long outputs; production is the reverse. A benchmark that reports 80 tokens a second is telling you about a shape of request you do not serve.

The uncomfortable bit

For an interactive product, a smaller model with a tight prompt usually beats a larger model with a generous one. Not because it is smarter. Because the product is experienced at the speed of its slowest part, and prompt length is the part you fully control.

A change that adds two eval points and 1.5 seconds to prefill is a regression. The eval does not sit in the chair. The user does.