Chunking Strategies That Change Your Answers

Chunking sets the ceiling on everything downstream

The vocabulary, defined tightly

Chunk

A passage of source text stored and indexed as a single retrievable unit, carrying its own identifier and metadata so it can be found, ranked and cited on its own.

Overlap

A number of tokens repeated at the boundary between neighbouring chunks, so a claim sitting across a cut still appears whole in at least one of them.

Parent-child retrieval

A pattern where small chunks are indexed and searched for precision, but the larger section containing the winning chunk is what gets sent to the model.

Contextual header

A short generated preamble attached to a chunk before embedding, naming the document, the heading trail and the situation the passage refers to, so the chunk reads correctly in isolation.

Late chunking

Embedding a long passage first so every token sees the full context, then pooling the token vectors into chunk vectors afterwards, rather than embedding each chunk in isolation.

The five splitting methods and what each one destroys

There are five methods in real use and they form a ladder of cost against fidelity. Most teams should start at structure-aware, not at fixed-size, because the extra day of work removes the largest category of unreachable facts.

The Chunk Contract: four clauses every chunk should satisfy

Stop asking how big a chunk should be and start asking what a chunk owes the rest of the system. Four obligations cover it, each with a test you can run on a random sample in an hour.

Overlap is insurance, not a strategy

Overlap repeats a slice of text at each boundary so a claim cut in half still appears whole somewhere. It is cheap insurance against the boundary problem and it is not a fix for bad boundaries, because it duplicates content into the index and duplicated content wins twice in the same result list.

Index small, send big

Precision and context pull in opposite directions, and you do not have to choose. Index a small unit so the vector is sharp and the match is specific, then expand to the parent section at assembly time so the model has enough surrounding material to reason with. The pattern goes by several names and it is the single biggest quality jump available after hybrid retrieval.

Three ways to expand

1. Sentence window. Index individual sentences, send the sentence plus a few either side. Best for dense reference material where the answer is one line and the neighbours qualify it.
2. Parent document. Index paragraph-sized chunks, send the whole section containing the winner. Best for policy and documentation, where the heading and the exceptions matter.
3. Summary index. Index a generated one-line summary of each section, send the full section. Best for long narrative documents where the literal wording matches nobody's question.

Expansion has a cost people forget: your context grows by the expansion factor, on every question, forever. Index at 150 tokens and send parents of 800 and your input tokens went up more than five times, which appears on the bill rather than in a metric. Cap the expansion and check the arithmetic in RAG explained for builders.

The contextual header, which is cheaper than it looks

Before embedding, prepend a block naming the document, the heading trail, the version and one generated sentence situating the passage. It costs one small call per chunk at ingest, once, and it fixes clause one for the whole corpus. It also improves keyword search, because the product name and version now sit in the chunk text where BM25 can see them.

# 1. The stored chunk format. Everything above the rule is generated
#    at ingest and prepended before embedding, so the chunk reads alone.

{doc_title} > {heading_path}
Version {version}, effective {effective_date}. Applies to: {audience}
Context: {situating_line}
---
{chunk_text}


# 2. The prompt that writes situating_line, one call per chunk at ingest.
#    Keep it short. This is a situating sentence, not a summary.

SYSTEM:
You are given a whole document and one passage from it. Write ONE sentence,
under 30 words, that lets a reader understand the passage without the
document. Resolve pronouns and vague references to their actual subject.
Name the entity, the product, the version and the time period the passage
is about, but only if the document states them. Do not add facts. Do not
summarise the passage. Do not use the words "this passage" or "the document".
If the passage is already self-explanatory, output the single word: NONE.

USER:
<document>{full_document}</document>
<passage>{chunk_text}</passage>

# Worked example
# passage:         "The window is 14 days from the invoice date."
# situating_line:  "Refund window for annual Enterprise plans bought after
#                   the January 2026 pricing change."

Tables, code and the documents that break every splitter

Some content cannot be cut by any general rule and needs a specific one. Tables are the worst offender, because a table split across chunks produces rows that have lost their column headers, and a row of numbers with no headers is worse than nothing. It is confidently retrievable and meaningless.

Sizing your corpus, with the arithmetic exposed

Before choosing a chunk size, run the numbers on your own corpus. The two figures that matter are how many vectors you will store, which drives index cost and query latency, and how many tokens each question will carry, which drives the recurring bill.

How to test a chunking change without fooling yourself

Change one thing, hold everything else frozen, and look at the questions that got worse rather than at the average. Chunking changes almost always improve the mean and almost always break a specific class of question, and the mean will happily hide that from you until a user finds it.

1. Freeze the retriever, the reranker and the modelnon-negotiable

   Same embedding model, same fusion weights, same top-k, same prompt, same model version. Two changes at once teaches you nothing and buys a week of arguing about which one did it.

2. Use a question set you did not generatethirty is enough

   Real user questions with hand-labelled correct sources. Model-generated questions about documents the model just read score well under any scheme, because they are phrased in the source's own words, which is exactly the case chunking does not need to handle.

3. Score recall@5 and recall@20 per question, not just the meankeep the table

   Store the per-question result for both runs side by side. The mean tells you the direction. The per-question table tells you the truth.

4. Read every question that went from found to missedthe real output

   This list is the actual result of the experiment. It usually clusters: every table lookup broke, or every question about exceptions broke because the exceptions now sit in their own chunk with no heading.

5. Check the citation surface, not only the hit rateeasily missed

   A smaller chunk can retrieve correctly and still produce a worse answer, because the model receives a fragment without the qualifying sentence. Read ten full answers, not just the retrieval scores.

6. Price the migration before you committhe part that stalls

   A chunking change means a full re-embed, a re-index and recalibrating every threshold including your abstention floor. Keep the old index until the new one is scored, and never swap the splitter and the embedding model in one release.

A default worth starting from

Here is a configuration that is hard to beat as a starting point for documentation, policies, contracts and knowledge base content. It is not a law and every clause of it should be measured on your own gold set. It exists so you can stop reading and start building today, then improve from a known position instead of from a blank page.

Once the split is right, the next failure to hunt is the one where retrieval works, the correct passage is in context, and the answer is still wrong. That is a different mechanism, covered in why your RAG system gives confident wrong answers.