How do you choose a chunking strategy for RAG?

Answer

Chunking determines what can be retrieved, so it caps the quality of everything downstream. A perfect retriever cannot fix a bad split.

Prefer semantic boundaries over fixed character counts. Split on document structure: sections, headings, paragraphs. For code, split on function or class boundaries. A chunk cut mid sentence retrieves badly because its embedding represents a fragment of an idea.

Size for self containment. A chunk must make sense alone, because that is how the model will see it. If understanding it requires the preceding paragraph, it is too small or needs its heading prepended.

Overlap modestly, commonly 10 to 20 percent, so an answer spanning a boundary is not lost. Too much overlap wastes storage and returns near duplicates that crowd the results.

Attach metadata. Source document, section title, date, and permissions. This enables filtering before similarity search, which is often more valuable than better embeddings. Restricting to the right document set beats ranking within the wrong one.

Consider what the query looks like. Retrieval matches query embeddings to chunk embeddings. If users ask short questions and chunks are long documents, the embeddings live in different regions of the space. Techniques such as embedding a generated summary or hypothetical question per chunk address this mismatch.

Evaluate empirically. Build a set of real questions with known correct sources, then measure whether the right chunk is retrieved. Without that, chunking changes are guesswork.

Common wrong answer

Picking a fixed token count because a tutorial used it, and never revisiting it. The right size depends on the documents and the queries. A legal contract and a chat log need different treatment, and 512 tokens is a starting guess, not an answer.