RAG Chunking Strategy: How to Split Documents So Retrieval Works

Your chunking strategy decides what your retrieval system is physically capable of finding, because a chunk is the smallest unit your index can return — if the answer to a question doesn’t sit inside one chunk, no embedding model, reranker, or top_k value will assemble it for you. The strategy that works on real documents is almost always the same shape: split on the document’s own structure rather than on a character count, keep the retrieval unit small and focused, and expand to a larger surrounding unit before you hand anything to the LLM. This guide is how I actually make those calls, including the parts the “five chunking strategies” listicles skip: what to do with tables and transcripts, what metadata has to travel with each chunk, and how to prove a chunking change helped instead of assuming it did.

The one-line answer

Chunk on structure, embed small, generate big.

Split documents at the boundaries the author already put there — headings, sections, list items, table rows, speaker turns — and let chunk length vary as a consequence. Index those focused units so the embedding actually concentrates on one idea. Then, at query time, expand each retrieved unit to its parent section before it goes into the prompt, so the model gets enough surrounding context to answer.

Everything else — token counts, overlap percentages, semantic splitters — is tuning on top of that. If you get the boundaries wrong, no amount of tuning saves you.

What chunking actually decides

It helps to be precise about what you’re choosing, because “chunk size” gets discussed as if it were one dial and it’s really four decisions:

1. The boundary rule. Where a chunk is allowed to start and end. This is the decision that matters most and gets the least attention.
2. The size target. How much text you aim to fit between boundaries.
3. The overlap. How much of the previous chunk you repeat at the start of the next one.
4. The unit you embed vs. the unit you send. Whether the thing you search over is the same thing the LLM reads. Most pipelines assume yes. It usually shouldn’t be.

The first and fourth are where the quality lives. The second and third are where teams spend their time.

There’s a physical reason boundaries dominate. An embedding is roughly an average of the meaning in the text it covers. A chunk that contains one clear idea produces a vector that points somewhere specific. A chunk that contains six loosely-related paragraphs produces a vector that points at the centroid of all of them — which is to say, nowhere in particular. That’s the dilution failure: the answer is in your index, it is technically retrievable, and it never ranks, because its signal was averaged into noise by everything else in the chunk.

The mirror failure is splitting. A fact and the condition it applies under land on opposite sides of a boundary. Now each half is individually unconvincing, and if one of them does retrieve, it’s actively misleading rather than merely useless — the model reads “customers get a 30-day window” without the sentence that said “enterprise contracts only.”

Both failures come from the same root cause: cutting on character counts, because character counts are blind to where an idea starts and stops.

Start from the question, not the token count

Before picking a splitter, answer one question about your corpus: what is the smallest unit of this document that can answer a real user question on its own?

That unit is your chunk. It’s a different thing for different corpora:

• API or product documentation: one section under one heading — the description, the parameters, and the example belong together. Splitting a code sample away from the sentence explaining it produces two useless chunks.
• Policies and contracts: one clause, with its heading path attached. Clause numbering is the retrieval unit users actually cite.
• Support tickets and email threads: one message, or one problem-and-resolution pair. Never the whole thread — a 40-message thread averaged into one vector matches everything and answers nothing.
• Meeting or call transcripts: a run of speaker turns on one topic. Fixed-size splitting on a transcript is uniquely destructive because it cuts mid-exchange, orphaning an answer from its question.
• Research papers and reports: a subsection. Abstracts are a special case worth indexing separately — they’re a dense summary that matches high-level queries the body never will.
• Tables and structured records: one row, serialized to text with its column names. More on this below.

If you can’t answer the smallest-unit question for your corpus, you don’t have a chunking problem yet — you have a “nobody has read a representative sample of these documents” problem. Read thirty. It takes an afternoon and it decides the next three months of retrieval quality.

Split on structure, then pack to a size

The implementation that follows from that: parse structure first, pack second.

Walk the document’s real hierarchy — markdown headings, HTML tags, PDF outline, XML nodes, whatever the format gives you — and produce a list of leaf sections. Then, for each leaf section:

• If it fits comfortably under your size target, that’s one chunk. Don’t merge unrelated small sections just to hit a token budget; a 40-token chunk that is one crisp definition is fine, and often excellent.
• If it’s larger than the target, split it internally on the next natural boundary down: paragraphs, then sentences. Never mid-sentence.
• Prepend the heading path to the chunk text so an isolated paragraph still carries the context that identifies it. Billing > Refunds > Enterprise exceptions in front of a paragraph changes both what it means to a reader and where it lands in embedding space.

That last step is disproportionately effective and costs nothing. A chunk that says “This does not apply to accounts on annual terms” is nearly unretrievable on its own. The same chunk prefixed with its section path is retrievable by anyone asking about annual billing terms.

Here’s the shape I actually implement:

import re
from dataclasses import dataclass, field

HEADING = re.compile(r"^(#{1,6})\s+(.*)$")

@dataclass
class Chunk:
    text: str            # what gets embedded
    heading_path: list[str]
    parent_id: str       # points at the section, for expansion at query time
    doc_id: str
    meta: dict = field(default_factory=dict)

def split_markdown(doc_id: str, markdown: str, target_tokens: int = 300,
                   count_tokens=lambda s: len(s) // 4):
    """Structure-first splitter: sections by heading, then paragraphs to fit.

    count_tokens defaults to a crude chars/4 estimate. Swap in your model's
    real tokenizer before you tune target_tokens — the estimate is fine for
    plumbing and wrong enough to matter for boundaries.
    """
    sections, path, buf = [], [], []

    for line in markdown.splitlines():
        m = HEADING.match(line)
        if m:
            if buf:
                sections.append((list(path), "\n".join(buf).strip()))
                buf = []
            level, title = len(m.group(1)), m.group(2).strip()
            path = path[: level - 1] + [title]
        else:
            buf.append(line)
    if buf:
        sections.append((list(path), "\n".join(buf).strip()))

    chunks = []
    for idx, (heading_path, body) in enumerate(sections):
        if not body:
            continue
        parent_id = f"{doc_id}#s{idx}"
        prefix = " > ".join(heading_path)

        # Pack paragraphs up to the target; never split a paragraph mid-sentence.
        current: list[str] = []
        def flush():
            if not current:
                return
            body_text = "\n\n".join(current)
            chunks.append(Chunk(
                text=f"{prefix}\n\n{body_text}" if prefix else body_text,
                heading_path=heading_path,
                parent_id=parent_id,
                doc_id=doc_id,
            ))
            current.clear()

        for para in re.split(r"\n\s*\n", body):
            para = para.strip()
            if not para:
                continue
            if current and count_tokens("\n\n".join(current + [para])) > target_tokens:
                flush()
            current.append(para)
        flush()

    return chunks

Two things about that code are load-bearing. The parent_id is what makes the next section possible. And the token counter is injected rather than hardcoded, because tuning target_tokens against a len(s) // 4 approximation and then deploying against a real tokenizer means your carefully-chosen 300 was never 300.

Decouple what you embed from what you send

This is the single highest-leverage move in chunking and most pipelines never make it.

The unit that retrieves best is small and focused — one idea, tight vector, high precision. The unit that answers best is larger — enough surrounding context that the model can reason without gaps. Those are different requirements, and there’s no size that satisfies both. So stop trying to find one.

Index the small unit. Store a pointer from it to its parent section. At query time, retrieve on the small units, then expand each hit to its parent (deduplicating when several children of the same section retrieve, which happens constantly) and send the parents to the LLM.

def expand_to_parents(hits, parent_store, max_parents=5):
    """Retrieve small, generate big. Dedupe: several children often share a parent."""
    seen, parents = set(), []
    for hit in hits:                       # hits are ranked; preserve that order
        if hit.parent_id in seen:
            continue
        seen.add(hit.parent_id)
        parents.append(parent_store[hit.parent_id])
        if len(parents) >= max_parents:
            break
    return parents

This one change fixes a whole category of complaints that get misdiagnosed as retrieval failures: “the right chunk came back but there wasn’t enough context to answer.” That’s not a ranking problem and reranking won’t touch it. It’s a unit-of-retrieval problem, and the fix is decoupling.

There’s a cheaper variant if a parent store is more machinery than you want: retrieve small, then pull the immediately adjacent chunks from the same document by position. Less principled, most of the benefit, about ten lines of code.

Size and overlap: honest defaults

Only after the boundary rule and the retrieval/generation split are settled do these numbers matter.

Size. For the embedded unit, 200–400 tokens is a reasonable starting band for prose. It’s large enough to carry a complete thought and small enough that the vector stays specific. Two hard constraints override it: never exceed your embedding model’s input limit (most silently truncate, so you’ll be indexing the first half of your chunk and never know), and never exceed what your reranker reads, if you have one — a cross-encoder that only sees the truncated head of a long chunk is scoring a document you didn’t give it.

Overlap. 10–15% of chunk size, and only when you’re splitting within a section. Overlap exists to reduce the odds that a fact and its qualifier are separated in every chunk. It is insurance, not a quality lever. Heavy overlap — the 50% you sometimes see recommended — inflates your index, makes near-duplicate results crowd out genuine diversity in the top-k, and papers over bad boundaries instead of fixing them. If you find yourself raising overlap to fix retrieval, the boundary rule is wrong.

When chunks come from structure — a whole clause, a whole table row, a whole ticket message — overlap is usually zero and should be. There’s nothing to bridge.

The document types that break naive splitting

Tables. Do not run a table through a text splitter. It will cut between rows and columns, and a chunk containing three cells with no headers is noise that pollutes your index. Serialize each row into a self-describing sentence — Plan: Enterprise | Seats: unlimited | SLA: 99.9% | Support: 24/7 — and index rows individually with the table caption prepended. For “compare the plans” queries, also index the whole table as one chunk. Yes, that duplicates content; the duplication is cheaper than the failure.

PDFs. The chunking is usually fine and the extraction is what’s broken. Two-column layouts get read across columns, interleaving two unrelated sentences. Headers and footers repeat into every chunk and drag every vector toward the same meaningless center. Before you touch chunk size on a PDF corpus, print ten extracted documents and read them. In my experience most “PDF RAG is bad” complaints are extraction bugs wearing a chunking costume.

Code and config. Split on syntactic units — function, class, block — not lines. A function body without its signature is unretrievable by anyone searching for the function. Keep imports and the enclosing class name in the prefix.

Transcripts. Group by topic shift or speaker exchange, keep speaker labels in the text, and attach timestamps as metadata. A transcript chunk without “who said this and when” is nearly worthless for the follow-up question, which is always “where did that come from.”

Semantic chunking, honestly

Semantic chunking embeds each sentence, measures similarity between neighbors, and cuts where the similarity drops — the idea being that the drop marks a topic shift.

It’s a real technique and I use it in one specific situation: unstructured prose with no usable structure. Scanned documents, transcripts without speaker markers, long-form writing with no headings. There, semantic splitting genuinely beats fixed-size, because there’s nothing better to cut on.

Everywhere else it loses to structure-first splitting, and it’s worth being clear about why. A heading is an explicit, authored statement that a new topic starts here. A cosine-similarity dip is a statistical guess that one might have. When the author already told you where the boundaries are, inferring them is strictly worse — plus it costs an embedding call per sentence at ingest, adds a threshold you now have to tune, and produces boundaries you can’t explain to anyone when retrieval goes wrong next quarter.

Reach for it when structure is genuinely absent. Don’t reach for it because the tooling makes it easy.

Prove the chunking change, don’t assume it

Chunking is where I most often see teams tune by superstition, because every change sounds reasonable and the feedback loop is a handful of spot-checked queries.

Build the measurement first. It’s the same eval loop that keeps every other retrieval decision honest, and I’ve written about it in more detail in the diagnostic guide for bad RAG retrieval:

1. Fifty real queries. From your logs if you have them, from your users if you don’t. Not queries you invented — invented queries flatter the system that produced them.
2. A gold document per query. Not a gold chunk — chunk identity changes every time you re-chunk, which makes chunk-level labels useless for exactly the comparison you’re running. Label at the document level and score a hit when any chunk from the gold document is retrieved.
3. Two metrics, not one. Recall at your retrieval depth (say @50) tells you whether the answer is reachable at all — that’s what chunking primarily controls. Recall or precision at the depth the LLM actually sees (@5 or @10) tells you whether the ranking survived. Chunking changes routinely improve one and hurt the other; a single number hides that.

Then change one thing and re-measure. Smaller chunks typically raise precision and fracture multi-sentence answers. Bigger chunks do the reverse. Structure-first boundaries usually raise both, which is why it’s the change I make first. The point of the harness isn’t the numbers — it’s that it tells you which of those tradeoffs you just took.

If reranking is in your pipeline, measure chunking before you add the reranker, not after. A reranker can only re-order what the first stage found, so it will mask a chunking regression at top-k while the recall ceiling quietly drops. The interaction is worth understanding properly; I go into it in the reranking guide.

Re-indexing is the real cost

The thing nobody tells you about chunking: it’s the one retrieval decision you can’t A/B cheaply, because changing it means re-embedding the entire corpus.

That has consequences you should plan for up front:

• Version your chunking config and store the version on every chunk. When retrieval quality shifts, “which chunking version produced this index” is the first question and you want it answerable in a query, not an archaeology project.
• Keep the raw document. Chunks are derived data. If your only copy of a document is its chunks, you can’t re-chunk without re-fetching, and the source may be gone.
• Make re-indexing a routine, boring operation — idempotent, resumable, and safe to run against a shadow index you can compare before cutting over. If a full re-index is a scary all-day event, you will avoid making chunking changes you should be making. The reliability properties that matter here are the same ones I describe in designing systems that survive disruption: partial failures that are visible, retries that don’t silently give up, and a path that resumes rather than restarts.
• Expect a shifted score distribution. Any threshold or cutoff tuned against the old chunks — a minimum similarity, a confidence gate — is meaningless against the new ones. Re-tune, or better, don’t ship absolute-score thresholds in the first place.

When you don’t need this

Some cases genuinely don’t need a chunking strategy, and it’s worth naming them before you build one.

Your documents are already short. FAQ entries, product records, support macros, catalog items. One document is one chunk. Splitting them is pure downside.

Your corpus fits in the context window. A few hundred pages of stable reference material, and the model can just read it — possibly with prompt caching so you’re not paying full price per call. Retrieval infrastructure earns its keep at scale or with churn, not at ten documents.

The question is aggregate. “How many,” “which customers,” “trend since March” — those are queries against structured data, not similarity searches over prose. No chunking strategy fixes an architecture mismatch; the fix is a query layer.

Retrieval isn’t your failing stage. If your gold documents are already retrieved and the answers are still wrong, chunking work is a detour. Localize first. The order of operations matters more than any individual fix, which is the whole argument of that diagnostic guide.

FAQ

What chunk size should I use for RAG? 200–400 tokens for the embedded unit is a sane default for prose, but treat it as a consequence of your boundary rule rather than a target to hit. If a natural section is 80 tokens, ship an 80-token chunk. The two hard limits are your embedding model’s input cap and your reranker’s context — exceed either and you’re silently indexing or scoring a truncated document.

How much overlap should chunks have? 10–15% when you’re splitting within a section, and zero when chunks come from real structural units. Overlap is insurance against a fact being separated from its qualifier in every chunk. If raising overlap measurably improves retrieval, that’s a signal your boundaries are wrong, not that you need more overlap.

Is semantic chunking better than fixed-size chunking? Usually yes, and it’s the wrong comparison. Both lose to structure-first splitting when the document has structure, because a heading is an authored boundary and a similarity dip is an inferred one. Semantic chunking earns its cost on genuinely unstructured prose — transcripts without speaker markers, scanned text, long-form writing with no headings.

Should I chunk by tokens or by characters? Tokens, using the tokenizer of the embedding model you’re actually deploying. Character counts drift badly across languages and across text with code, identifiers, or heavy punctuation — exactly the content where a truncated chunk hurts most. A chars / 4 estimate is fine for plumbing and wrong enough to matter once you’re tuning.

Do I need to re-index everything when I change chunking? Yes. Chunk boundaries determine the embeddings, so new boundaries mean new vectors for every affected document. Plan for it: version the chunking config on each chunk, keep the raw documents, and make re-indexing a resumable operation against a shadow index so you can compare before cutting over.

Can a reranker compensate for bad chunking? No. A reranker re-orders what the first stage returned — if a chunk boundary destroyed the answer, there’s nothing in the candidate set to promote. Reranking fixes ranked-too-low; chunking fixes not-retrievable-at-all. They’re different failures with no fixes in common.

Chunking is the first place where a retrieval system’s ceiling gets set, which is why I treat it as an architecture decision rather than a config value. The rest of that stack — hybrid fusion for the queries where exact tokens matter, reranking for precision, and the evaluation loop that keeps all of it honest — is collected in my guides on hybrid search and RAG, and the tradeoff between lexical and semantic matching that chunk size quietly interacts with is laid out in hybrid search vs vector search. Turning a retrieval stack that returns plausible-but-wrong results into one you can measure and improve on purpose is the work I do in relevance and correlation scoring.