How to Split Documents for RAG
Chunk boundaries decide what can be retrieved at all. Too small loses context, too large dilutes relevance, and structure beats character counts.
On this page
Chunking looks like a preprocessing detail. It is one of the two biggest determinants of RAG quality, alongside reranking.
The reason is blunt: a chunk is the unit of retrieval. If the answer to a question spans two chunks, neither chunk contains the answer, and no amount of prompt engineering recovers it.
The tension
Small chunks produce precise embeddings — one focused idea per vector, so similarity scores mean something. But they lose surrounding context. A chunk reading “This is not recommended for production” is useless without knowing what this is.
Large chunks preserve context but blur meaning. A 2,000-word chunk covering six topics produces an embedding that is the average of six things, matching every related query weakly and none strongly. It also wastes context window on irrelevant text.
There is no universally correct size. There is a correct approach, and it is to stop thinking in character counts.
Split on structure first
The most common mistake is splitting every 1,000 characters regardless of content. That cuts mid-sentence, mid-table, mid-function.
Documents already have meaningful boundaries. Use them.
Markdown and HTML have headings. A section under one heading is usually one coherent idea — that is why the author put a heading there. Split on headings, and only subdivide sections that exceed your size limit.
Code has functions, classes, and modules. Splitting a function in half produces two useless chunks. Syntax-aware splitting respects these boundaries.
PDFs are the hard case. Extraction often destroys structure before chunking begins, and a mangled table produces noise that flows through every later stage. Fix extraction before tuning chunk size — this is where more RAG problems originate than people expect.
Transcripts and chat logs split on speaker turns or topic shifts, not fixed lengths.
Recursive splitting is the practical fallback: try paragraph breaks, then sentence breaks, then words, descending only when a chunk is still too large. Most libraries implement this, and it is a reasonable default when documents lack usable structure.
Overlap
Include the last portion of each chunk at the start of the next — commonly 10–20%.
This softens boundary damage. A sentence spanning a split appears complete in at least one chunk. Cheap insurance against the worst failure mode.
The cost is storage and some duplicate retrieval, where two overlapping chunks both match and consume context saying nearly the same thing. Deduplicating overlapping neighbours before assembling the prompt is worth doing.
Add context back
If small chunks lose context, put the context back explicitly. Several techniques, roughly in order of effort:
Prepend the breadcrumb. Store the document title and heading path with each chunk: Product Docs > Billing > Refunds: <chunk text>. Trivial to implement, and it substantially improves both retrieval and the model’s ability to interpret what it received.
Retrieve small, send large. Embed and match on small precise chunks, but send the surrounding parent section to the model. You get precision in retrieval and context in generation. This resolves the core tension more cleanly than any size tuning.
Add a generated summary. Have a cheap model write one sentence of context per chunk at index time and store it alongside. Effective, and it costs one indexing pass.
Store neighbour links. Keep pointers to adjacent chunks so you can expand a match into its surroundings on demand.
Metadata
Store generously at index time: source document, section path, date, author, permissions, URL, document type.
You need it for filtering (only current documents, only what this user may see), for citations, and for debugging. Adding it later means re-indexing everything, so err toward too much.
Practical starting points
A reasonable default for prose: structure-aware splitting, target around 500–1,000 tokens per chunk, 10–15% overlap, breadcrumb prepended.
Then tune against measurements rather than intuition. The measurement that matters: for a set of real questions with known answers, was the correct chunk in the retrieved set at all? That single number tells you whether chunking is your problem. If the right chunk never gets retrieved, adjusting your prompt is wasted effort.
Different content types in one corpus usually warrant different strategies. Code, prose documentation, and support tickets do not chunk the same way, and forcing one strategy across all of them costs quality.
What to remember
- A chunk is the unit of retrieval — answers spanning boundaries become unretrievable.
- Small chunks give precise embeddings but lose context; large chunks blur meaning and waste context.
- Split on structure (headings, functions, turns), not fixed character counts; recursive splitting is a good fallback.
- Overlap 10–20% to soften boundary damage, then deduplicate overlapping matches.
- Retrieve small, send large and prepending breadcrumbs both resolve the size tension directly.
- Measure whether the correct chunk was retrieved before tuning anything else.