What is RAG and when do you need it?

Answer

RAG, retrieval augmented generation, means fetching relevant material and including it in the prompt so the model answers from it rather than from training data alone.

The pipeline: split documents into chunks, embed them, store the vectors, embed the query at request time, retrieve the closest chunks, put them in the prompt, generate.

You need it when the model must answer from knowledge it does not have: internal documents, recent events after the training cutoff, or user specific data. It also gives you citations, since you know which chunks were used.

Where it is genuinely valuable is when the corpus is large, changes frequently, and needs attribution.

Where it is over applied is more common. If the corpus is small enough to fit in context, just put it in context. Modern windows hold a lot, and skipping retrieval removes an entire failure mode. Retrieval only helps when you cannot include everything.

Agents complicate the picture. An agent with search tools performs retrieval dynamically, and can search several times, refine its query, and follow leads. That is often better than one-shot vector similarity, because relevance is judged by a model rather than by cosine distance.

Common wrong answer

Treating RAG as a fix for hallucination. It reduces some hallucination by grounding answers in real text, but the model can still misread a retrieved chunk, combine two sources incorrectly, or answer confidently when retrieval returned nothing useful. Retrieval quality becomes the new failure point rather than eliminating failure.

Also common is assuming RAG requires a vector database. Keyword search, SQL filters, or a plain file read all count as retrieval, and are frequently better for structured or exact match queries.