Retrieval failures I keep seeing

4 November 2026 · Bristol

Chunk size gets all the attention and almost none of the incidents. Here is what actually took systems down in the last three I looked at, in the order I now check them. One of these was mine.

Nobody owns freshness

An index is a cache with a truth problem. Somebody has to be able to answer "how old is the oldest document a user can retrieve right now" with a query, not a guess. In the worst case I saw, a connector had been silently failing since a credential rotation. Seven weeks. The dashboard was green because it was measuring query latency, not ingestion.

So I now ask for one metric: lag from source change to searchable, p95. If it is not instrumented, that is the first ticket.

Duplicates read as consensus

Five copies of the same paragraph do not just waste tokens. They outrank the one document that actually answers the question, and the model sees the same claim five times, which looks like agreement. I have watched a system confidently repeat a policy that had been superseded because three stale copies of the old version beat one current page.

A shingle hash at ingestion and a canonical source id per cluster fixes most of it. Cheap, unglamorous, rarely done.

Embedding drift, which was my bug

I upgraded the embedding model to get better recall on short queries. Forgot that the existing collection was built with the previous one. Nothing errored. Searches still returned results, they were just worse in a way that took six weeks to attribute, because the prompt and the index had changed in the same month.

Now the index stores the model id and dimension, and refuses to serve a query from a different encoder. A hard failure would have cost me an afternoon. The silent version cost six weeks of confused customers.

Measure retrieval on its own

Answer quality is a lagging indicator. It moves when the prompt changes, when the model changes, when the user population changes. Keep forty queries with known correct documents and measure recall at 5. When answers get worse, that number tells you within a minute whether the index is at fault.

Forty is not a statistically rigorous sample. It has been enough to catch every index regression I have hit, and it takes an hour to build.

The context budget is a budget

More retrieved text is not more truth. Every extra document adds prefill, adds latency, and adds a chance to distract the model with something plausible and irrelevant. Three documents plus a reranker beat twenty without one in both systems where I measured it, at about a fifth of the cost.

If you take nothing else from this page: the failures are almost never in the part of retrieval that people write blog posts about.