Chunking Strategies for RAG (and Anthropic's Contextual Retrieval): The Engineer's Guide

The short answer: recursive 512-token chunks, then contextualize

Start with recursive 512-token chunks using RecursiveCharacterTextSplitter and 10-20% overlap, matching chunk size to your query type. For the biggest single quality jump, adopt Anthropic’s Contextual Retrieval: prepend a Claude-generated 50-100-token context to each chunk before you embed and BM25-index it. With hybrid search and reranking on top, that cuts retrieval failures by up to 67%.

Everything below is the engineering detail behind those three sentences.

Chunking is the upstream lever nothing downstream can recover from

Retrieval-augmented generation has a pipeline shape: split documents into chunks, embed them, store them, retrieve the top matches for a query, and hand them to the model. Most teams spend their tuning budget on the downstream end — a fancier embedding model, a reranker, a bigger vector index. That’s backwards.

If a chunk is split badly — a table cut in half, a definition severed from the term it defines, a “3% growth” figure orphaned from the quarter it describes — no reranker and no better embedding model can recover the information that split destroyed. It was never in the index to begin with. How you split decides the ceiling on what retrieval can ever find. Chunking is the one lever where a bad default silently caps every metric you’ll later try to improve.

That’s why this is the first thing to get right when you build a RAG system with Claude, and why it deserves more than the throwaway chunk_size=1000 most tutorials hand you.

The strategies: fixed-size, recursive, semantic, and structure-aware

There are four workhorse chunking strategies before you get to Anthropic’s contextual technique, and they trade quality against cost:

• Fixed-size splits at a fixed token or character count with a sliding window and no awareness of meaning. It’s fine for prototyping and homogeneous text, but it will happily cut a sentence — or a word — in half.
• Recursive (RecursiveCharacterTextSplitter) splits on a hierarchy of separators: paragraph, then sentence, then word. It only falls back to a smaller separator when a chunk is still too big, so it respects natural boundaries far more than fixed-size. This is the recommended baseline workhorse.
• Semantic splits where meaning actually shifts, using embedding similarity between adjacent segments to detect topic boundaries. Higher quality, but roughly 14x slower than token-based splitting because it embeds while it splits.
• Structure-aware splits on the document’s own structure — Markdown headings, sections, HTML tags. Excellent for well-formatted docs where the author already marked the boundaries for you.

The rule of thumb: start recursive, and only graduate when your retrieval metrics justify the extra cost. Here is the baseline every RAG system should start from. Note the token-accurate length function — character counts drift badly from token counts on real text, and your embedding and context budgets are measured in tokens.

import tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Token-accurate length so chunk_size means TOKENS, not characters.
enc = tiktoken.get_encoding("cl100k_base")

def token_len(text: str) -> int:
    return len(enc.encode(text))

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,          # baseline: 512 tokens
    chunk_overlap=64,        # ~12.5% overlap, inside the 10-20% band
    length_function=token_len,
    separators=["\n\n", "\n", ". ", " ", ""],  # paragraph -> line -> sentence -> word
)

chunks = splitter.split_text(document_text)
print(f"{len(chunks)} chunks, avg {sum(token_len(c) for c in chunks) // len(chunks)} tokens")

The separators list is the recursion: the splitter tries to break on blank lines first, and only descends to sentences, spaces, and finally raw characters when a piece is still over budget. That’s what keeps paragraphs and sentences intact.

Chunk size and overlap: match the size to your query type

Sizing is where most of the easy wins live:

• Start at 512 tokens. A February 2026 benchmark ranked recursive 512-token chunks first overall. It’s a strong default across mixed corpora.
• 512-1024 tokens is a defensible range. Smaller chunks give precise matches but lose surrounding context; larger chunks carry more context but dilute the embedding and pull in noise.
• Use 10-20% overlap. Overlap is a safety margin: a fact sitting near a chunk boundary appears in both neighbors, so a query can match it from either side. Overlap recovers roughly 60-70% of boundary loss at minimal extra storage.
• Count tokens, not characters. Use tiktoken (or your embedding provider’s tokenizer — see the Voyage AI tokenizer docs) so chunk_size means what you think it means.

The old “always add overlap, always use 1000 characters” advice is no longer safe to assume. Chunking choice alone can swing Recall@k by up to roughly 9% on the same corpus — that’s a bigger delta than swapping embedding models on many datasets. The nuance is that the right size depends on your query type, not a universal constant. Short factoid queries (“what is the API rate limit?”) want small, precise chunks so the match isn’t diluted. Broad, synthesis-style queries (“summarize the refund policy”) want larger chunks that carry a whole argument. There is no single number that wins for both — which is exactly why you need an eval harness to measure retrieval on your queries before you commit. Pick the size your metrics reward, not the one a tutorial hard-coded.

Anthropic Contextual Retrieval: the problem a naked chunk can’t solve

Here’s the failure mode that better sizing can’t fix. Suppose a chunk reads:

The company’s revenue grew by 3% over the previous quarter.

Split out of its document, that sentence is nearly useless to retrieval. Which company? Which quarter? A query like “what was Acme’s Q2 2025 revenue growth?” may never surface this chunk, because the chunk’s own text contains neither “Acme” nor “Q2 2025.” The embedding and the keyword index both index what’s in the chunk — and the identifying context lives in a heading three paragraphs up.

Anthropic’s Contextual Retrieval fixes this at index time. Before embedding, you prepend a short, chunk-specific context blurb — usually 50-100 tokens — generated by Claude, that situates the chunk in its parent document. The orphaned sentence becomes:

This chunk is from Acme Corp’s Q2 2025 earnings report, discussing quarterly financial performance. The company’s revenue grew by 3% over the previous quarter.

Now both the embedding and the keyword index carry the missing “which company / which quarter.” Two variants are used together: Contextual Embeddings (contextualize before you embed) and Contextual BM25 (contextualize before you build the keyword index).

This is the core call. The critical detail is prompt caching: you load the whole reference document into the cache once, then reuse it across every chunk you contextualize, instead of re-sending the full document per chunk. That’s what makes the technique affordable.

import anthropic

client = anthropic.Anthropic()

DOC_CONTEXT_PROMPT = "<document>\n{doc}\n</document>"

CHUNK_CONTEXT_PROMPT = """Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""

def contextualize(document: str, chunk: str) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": DOC_CONTEXT_PROMPT.format(doc=document),
                    "cache_control": {"type": "ephemeral"},  # cache the whole doc once
                },
                {
                    "type": "text",
                    "text": CHUNK_CONTEXT_PROMPT.format(chunk=chunk),
                },
            ],
        }],
    )
    return resp.content[0].text

# Prepend the generated context, THEN embed and BM25-index the result.
contextualized = [f"{contextualize(document_text, c)}\n\n{c}" for c in chunks]

The cache_control marker on the document block is the whole game: the first chunk pays to write the document into the cache; every chunk after reads it at a fraction of the price. See the prompt caching docs for the read/write pricing. If you’re new to the SDK, start with how to use the Claude API.

The verified numbers: 35% to 49% to 67% fewer retrieval failures

Anthropic measured this against a 5.7% baseline retrieval failure rate (using top-20-chunk retrieval), and the gains stack:

• Contextual Embeddings alone reduce retrieval failures by 35% (5.7% to 3.7%).
• Adding Contextual BM25 brings it to 49% (5.7% to 2.9%) — this is hybrid search: semantic embeddings and keyword BM25 together, both contextualized.
• Adding a reranking step on top reaches 67% (5.7% to 1.9%). The reranking step retrieves the top-150 chunks, reranks them, and keeps the top-20.

The reranking half of that stack — hybrid search plus a reranker, and how to measure the lift on your own data — is the subject of the reranking and hybrid search sibling post. This post owns the upstream half.

The obvious objection is that running Claude over every chunk sounds expensive. It isn’t, because of prompt caching: since you load each reference document into the cache once and reuse it for all its chunks, Anthropic reports a one-time cost of $1.02 per million document tokens to generate the contextual blurbs. That’s an indexing cost, not a per-query cost — you pay it when you build the index and never again at query time. For a corpus of a few million tokens, that’s a couple of dollars to meaningfully raise your retrieval ceiling. Weighed against the engineering hours you’d otherwise sink into chasing a 9% recall swing by hand, it’s one of the best-value moves in the whole pipeline.

A decision guide: start simple, graduate only when your metrics justify it

Don’t reach for the fanciest strategy on day one. The right escalation path:

The gate at every step is the same: your retrieval metrics. Semantic chunking is ~14x slower and Contextual Retrieval adds an indexing cost — you only pay those if an eval on your corpus proves they help. Grounding your generation on well-retrieved context is also how you build an AI chatbot without hallucinations: the model can’t cite what retrieval never found.

For quick reference on where each strategy fits: fixed-size for prototypes and uniform text; recursive as the default for almost everything; structure-aware for Markdown and well-sectioned docs; semantic when topic boundaries matter and you can afford the ~14x cost; Contextual Retrieval when retrieval quality is the product and orphaned context is your failure mode.

Next: reranking, hybrid search, and how to measure any of this

Chunking sets the ceiling; the rest of the pipeline decides how close you get to it. From here, three directions: add the reranking and hybrid search layer that turns the 49% into 67%, stand up the minimum eval harness so every change above is a measured decision rather than a guess, and lean on prompt caching to keep contextualization near-free. All of it is part of the broader AI agents toolkit. Start with recursive 512, measure, then contextualize — in that order.