Building a basic RAG pipeline
Build the smallest version first. It will answer questions today, and it will tell you where the real problems are.
The whole pipeline
Two phases: index once, then answer many times.
Indexing
- Load documents.
- Split into chunks, on structure where possible.
- Embed each chunk.
- Store the vector, the text, and the metadata.
Answering
- Embed the question.
- Find the closest chunks.
- Put them in a prompt with the question.
- Generate the answer.
The prompt is where correctness is decided
This is the step people rush, and it is the one that decides whether your system lies.
Answer the question using only the context below. If the context does not contain the answer, say "I do not have information about that" and stop. Do not use outside knowledge. Cite the source of each claim using its number. Context: [1] (billing.md, Refunds) Refunds are processed within five working days... [2] (billing.md, Fees) Transaction fees are non refundable... Question: How long do refunds take?
Three things are doing work here. Restricting to the context stops invention. The explicit out gives it permission to fail rather than guess. The citation requirement makes answers checkable, and it makes retrieval failures visible instead of silent.
Start without a vector database
For a few thousand chunks, keep vectors in memory and compute cosine similarity in a loop. It is fast enough and there is nothing to operate.
Add a real vector store when you have measured that you need one, or when you need the index to persist and be shared. Qdrant is a reasonable choice when that day comes.
Retrieve a few, not many
Three to five chunks is a good default. More is not better: it dilutes the context, costs more, and drags in near misses that mislead.
If three good chunks do not answer the question, the fix is better retrieval, not more chunks.
Log everything from day one
For each question, record the question, which chunks came back with their scores, and the final answer.
Without this you cannot tell a retrieval failure from a generation failure, and they need completely different fixes. This is the most commonly skipped step and the one you will most regret skipping.
Add these only when measurement says so
Hybrid search. Add when exact identifiers or product codes are being missed.
Reranking. A second model reorders the top twenty into a better top five. Add when the right chunk is being retrieved but ranked too low to make the cut.
Query rewriting. Expand or rephrase the question before searching. Add when users write terse or ambiguous queries.
Metadata filtering. Restrict by date, product, or permission before searching. Add as soon as you have multiple tenants or versions, because this one is a correctness issue rather than a quality one.
The order that works
Get the simple version answering real questions. Log it. Look at the failures. Then fix the specific thing that is actually broken.
Building the sophisticated version first means you cannot tell which of your five clever components is the one causing the bad answer.