When RAG fails

A RAG system that gives a bad answer failed at one of two steps. Diagnosing which one comes first, because the fixes share nothing.

The first question

Look at what retrieval returned.

If the correct passage was not in the results, that is a retrieval failure. If it was there and the answer is still wrong, that is a generation failure.

You can only do this if you log retrieved chunks. If you are not logging them, start there.

Retrieval failures

Vocabulary mismatch. The user's words and the document's words differ, and semantic search did not bridge the gap. Fix with hybrid search, or by adding common phrasings to the document.

Exact identifiers. Error codes, SKUs, version numbers. Embeddings handle these badly because they carry no semantic meaning. Fix with keyword search running alongside.

Bad chunking. The answer was split across a boundary, so no single chunk contains it. Fix with structural splitting and overlap. See chunking and indexing.

Ranked too low. The right chunk was retrieved at position twelve and you take the top five. Fix with reranking, not by taking more chunks.

It is not a search question. "How many X last month" needs aggregation. No amount of retrieval tuning fixes this. Route it to a query instead.

Generation failures

Invented detail. The context was right, the answer added things not in it. Fix by restricting the prompt to the context explicitly, and by requiring citations so the invention has nowhere to hide.

Wrong passage chosen. Five chunks retrieved, the model used the least relevant one. Usually means the chunks were too similar. Improve ranking so the best one is clearly first.

Refuses despite having the answer. Over strict prompting. Soften from "only if explicitly stated" to "using the context, which may require combining two passages."

Contradictory sources. Two documents disagree, often because one is outdated. The model picks arbitrarily. Fix at the data layer: filter by date, or remove the stale document. No prompt resolves a genuine contradiction well.

The failure that matters most

Confident answers from wrong context.

The model does not know retrieval failed. Given three irrelevant passages, it will write a fluent, well structured, wrong answer with the same tone it uses when correct.

This is why citations matter so much. A cited answer can be checked in seconds. An uncited one has to be trusted.

Measure retrieval separately

Build a set of twenty to fifty real questions with known correct source passages. Then measure one thing: how often is the correct passage in the top few results.

This number tells you whether to work on retrieval or on generation. Without it you are tuning prompts to compensate for a search problem, which never quite works.

The honest limit

Some questions cannot be answered from your documents because the answer is not in them. The correct behaviour is to say so.

A system that says "I do not have information about that" is more useful than one that always produces something. Design for that answer, and test that it happens.