Why your RAG retrieves the wrong chunks (and how to prove it)

If your RAG returns plausible-but-wrong passages, the fix is a two-stage retrieval pipeline. Run dense (embedding) search and BM25 keyword search in parallel, fuse the two ranked lists with Reciprocal Rank Fusion (RRF, k=60) into a wide top-100, then rerank those with a cross-encoder like Voyage rerank-2.5 down to the top-10 you hand to Claude. Then measure Recall@k, MRR, and NDCG before versus after, and keep only the changes your numbers justify.

This post builds on the base pipeline from build a RAG system with Claude; here we make its retrieval accurate and prove the lift. The trap is trusting a demo. Dense-only top-k feels right on the queries you tested by hand, then quietly fails on the long tail in production. You cannot see that failure without a labeled eval set, which is why measurement is the spine of this post, not an afterthought at the end. If you have not decided whether RAG is even the right tool, read RAG vs fine-tuning first.

Why dense-only top-k fails: exact terms and similar-but-not-relevant

Plain embedding search misses in two distinct ways, and they compound.

First, it underweights exact-match terms. Embeddings map text into a semantic space, which is exactly why a product code like SKU-4471-B, a rare acronym, or a person’s surname gets smeared into a fuzzy neighborhood instead of matched literally. The user typed the one token that uniquely identifies the answer, and the vector search shrugged.

Second, it returns similar-but-not-relevant chunks. Cosine similarity rewards topical closeness, so a passage about “how to reset a password” scores high for the query “reset my billing PIN” — same shape, wrong answer. The right chunk is in your corpus; it just is not in the top-k the bi-encoder returned.

Misses exact terms

Product codes, SKUs, part numbers
Rare acronyms and proper nouns
The literal token that pins the answer
BM25 keyword search fixes this

Returns similar-but-not-relevant

Topically close, factually wrong
High cosine, low true relevance
Right chunk exists but ranks below the cutoff
A cross-encoder reranker fixes this

Two fixes stack cleanly onto these two failures: hybrid search raises recall (gets the right chunk into the candidate set at all), and reranking raises precision (pushes it to the top so it survives the cut to top-10).

Hybrid search: dense + BM25, and when each one wins

Hybrid search means running two retrievers and fusing their results. The dense retriever is the Voyage embedding search you already built. The sparse retriever is BM25, the classic keyword ranking function that scores documents by term frequency and rarity — the same family that powers Elasticsearch and Postgres full-text search.

They win on opposite queries. BM25 nails the exact-term queries embeddings fumble: give it SKU-4471-B and it finds the one chunk containing that literal string. Dense search nails the paraphrases BM25 fumbles: ask “how do I stop the app from crashing on launch” and it retrieves the chunk about “resolving startup exceptions” even though they share no keywords. Run both, and each covers the other’s blind spot.

The point of hybrid is recall, not final ranking. You are not trying to perfectly order results yet — you are trying to make sure the right chunk lands somewhere in the merged candidate pool so the reranker can promote it. A typical setup pulls BM25 top-50 and dense top-50, then fuses.

Reciprocal Rank Fusion (RRF): the score(d) = sum 1/(k+rank) formula, k=60

The problem with merging two ranked lists is that their scores are incompatible. BM25 scores are unbounded term-frequency sums; cosine similarities live in a bounded range. Averaging them is meaningless. Reciprocal Rank Fusion sidesteps this by ignoring the raw scores entirely and fusing on ranks only.

For each document d, RRF sums a small contribution from every list it appears in, where the contribution shrinks with rank position:

def rrf(ranked_lists, k=60):
    # ranked_lists: list of lists, each an ordered list of doc IDs (best first).
    scores = {}
    for lst in ranked_lists:
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    # Sort by fused score, highest first.
    return sorted(scores, key=scores.get, reverse=True)

# Fuse BM25 and dense results into one ranked candidate list.
fused = rrf([bm25_ids, dense_ids], k=60)
candidates = fused[:100]   # keep a wide top-100 for reranking

The formula is score(d) = sum over lists of 1 / (k + rank(d)). The constant k=60 is the common default: it damps the influence of the very top ranks so a single list cannot dominate, and it makes the fusion robust to noisy top positions. A document that appears near the top of both lists accumulates the most, which is exactly the consensus signal you want. RRF is cheap, has no scores to normalize, and beats naive weighted-score blending precisely because it never touches the incompatible scales.

Reranking: cross-encoder vs bi-encoder, retrieve-wide-rerank-narrow

Hybrid search gets the right chunk into the candidate set. A reranker is what pushes it to the top. The distinction that matters is bi-encoder versus cross-encoder.

Your first-stage retrieval uses a bi-encoder: it embeds the query and each document separately, then compares vectors. That is fast and precomputable — you can index millions of document vectors ahead of time — but the query and document never actually “see” each other, so the relevance signal is coarse.

A cross-encoder reranker reads each (query, passage) pair jointly in one forward pass and scores true relevance. It is dramatically more accurate because it can model the interaction between the two texts directly. The catch is cost: you cannot run a cross-encoder over your whole corpus, only over a shortlist. So the pattern is retrieve wide, rerank narrow — fuse to a top-100, rerank those 100, and keep the top-10 for the LLM. On the MTEB and BEIR benchmarks, adding a cross-encoder reranker typically yields a +5 to +15 NDCG@10 lift over first-stage retrieval alone.

BM25 top-50sparse keyword recall (exact terms)
Dense top-50bi-encoder semantic recall (paraphrases)
RRF merge -> top-100rank fusion, k=60, wide candidate pool
Cross-encoder rerank -> top-10joint (query, passage) scoring for precision
Generate with Claudegrounded on the 10 best chunks

Which reranker: Voyage rerank-2.5, Cohere, BGE

Anthropic recommends Voyage AI for both embeddings and reranking, and it slots in with the same client you already use for embeddings. The current models are rerank-2.5 and the faster rerank-2.5-lite. Per Voyage’s own numbers, they improve retrieval accuracy +7.94% and +7.16% respectively over Cohere Rerank v3.5, support a 32K-token context (so you can rerank long chunks without truncation), and are instruction-following — you can pass an instruction alongside the query to steer what “relevant” means for your domain.

Calling it is one line. You pass the query and the candidate texts, and it returns them re-scored and re-ordered:

import voyageai

vo = voyageai.Client()  # reads VOYAGE_API_KEY from the environment

reranked = vo.rerank(
    query="how do I reset my billing PIN?",
    documents=candidate_texts,     # the ~100 texts from the RRF merge
    model="rerank-2.5",
    top_k=10,                      # keep only the 10 best for Claude
)

best_chunks = [r.document for r in reranked.results]
# reranked.results are ordered best-first, each with .relevance_score

The alternatives are real and worth knowing: Cohere Rerank (3.5 / 4 Pro) is a hosted competitor, and open-source BGE and Jina cross-encoders (plus FlashRank for a lightweight local option) let you self-host with no per-call fee. The mechanism is identical across all of them — a cross-encoder scoring (query, passage) pairs — so you can swap the reranker without touching the rest of the pipeline. See the Voyage reranker docs for the full API.

The full pipeline: BM25 + dense, then RRF, then rerank, then Claude

Here is the whole two-stage retrieval assembled. First-stage retrieval runs BM25 and dense in parallel; RRF fuses them to a wide pool; the cross-encoder reranks to the final ten; Claude generates a grounded answer over those ten.

import voyageai
import anthropic

vo = voyageai.Client()
claude = anthropic.Anthropic()

def retrieve(question, corpus, bm25_index, doc_embeddings):
    # 1. Sparse: BM25 keyword search -> top-50 doc IDs.
    bm25_ids = bm25_index.top_n(question, n=50)

    # 2. Dense: bi-encoder vector search -> top-50 doc IDs.
    q_vec = vo.embed([question], model="voyage-4",
                     input_type="query").embeddings[0]
    dense_ids = nearest_neighbors(q_vec, doc_embeddings, n=50)

    # 3. Fuse both lists with RRF (k=60) and keep a wide top-100.
    candidate_ids = rrf([bm25_ids, dense_ids], k=60)[:100]
    candidate_texts = [corpus[i] for i in candidate_ids]

    # 4. Cross-encoder rerank -> top-10 for the LLM context.
    reranked = vo.rerank(query=question, documents=candidate_texts,
                         model="rerank-2.5", top_k=10)
    return [r.document for r in reranked.results]

def answer(question, chunks):
    # 5. Generate a grounded, cited answer with Claude.
    context = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(chunks))
    resp = claude.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system=("Answer using ONLY the numbered context. Cite as [n]. "
                "If it's not there, say you don't know.\n\n" + context),
        messages=[{"role": "user", "content": question}],
    )
    return resp.content[0].text

The nearest_neighbors and bm25_index calls stand in for whatever store you use — swap in the pgvector cosine-distance query and a Postgres or Elasticsearch full-text index from vector databases compared. The shape of the pipeline does not change. Step 5 is the Claude API answering with grounding.

Measuring the lift: Recall@k, MRR, NDCG, and a before/after table

This is the whole point: prove the lift, do not assume it. Every change above costs latency and money, so you only keep the ones your metrics justify. The retrieval metrics you care about are:

Recall@k / Hit-Rate@k — is the known-relevant chunk anywhere in the top-k? This is the recall question hybrid search targets.
Precision@k — what fraction of the top-k are actually relevant?
MRR (mean reciprocal rank) — how high does the first relevant hit land, averaged over queries? This is what reranking moves.
NDCG@k — rank-quality, rewarding relevant chunks for being near the top.

Here is a self-contained snippet for the two that matter most, run over a labeled eval set:

def recall_at_k(retrieved_ids, relevant_id, k):
    return 1.0 if relevant_id in retrieved_ids[:k] else 0.0

def reciprocal_rank(retrieved_ids, relevant_id):
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id == relevant_id:
            return 1.0 / rank
    return 0.0

def evaluate(eval_set, retrieve_fn, k=10):
    recalls, rrs = [], []
    for query, relevant_id in eval_set:   # eval_set: (query, known_relevant_chunk_id)
        ids = retrieve_fn(query)
        recalls.append(recall_at_k(ids, relevant_id, k))
        rrs.append(reciprocal_rank(ids, relevant_id))
    return {"recall@k": sum(recalls) / len(recalls),
            "mrr": sum(rrs) / len(rrs)}

before = evaluate(eval_set, dense_only_retrieve)      # baseline
after  = evaluate(eval_set, hybrid_rerank_retrieve)   # new pipeline

Run it on your baseline and your new pipeline, and put the two rows next to each other. The RAGAS framework is the open-source standard for the generation side — it defined the four-metric pattern of context precision, context recall, faithfulness, and answer relevance — so once retrieval is solid, measure whether the answers stayed grounded too. For the broader harness, see the minimum eval setup for an AI agent.

Recall@10 — dense-only baselinebaseline

Recall@10 — hybrid + rerankhigher recall

MRR — dense-only baselinebaseline

MRR — hybrid + rerankfirst hit ranks higher

Building a small labeled eval set (query to known-relevant chunk)

You do not need thousands of labels to start. A useful eval set is 30 to 50 queries, each paired with the ID of the chunk that actually answers it. Draw the queries from real logs, real support tickets, or the questions your team already knows users ask — the exact-term and paraphrase cases from earlier belong here on purpose, because those are where dense-only breaks.

Label them once, store them as a plain list of (query, relevant_chunk_id) tuples, and re-run evaluate() on every pipeline change. That is your regression harness: when someone proposes bumping the RRF pool to top-200 or swapping the reranker, the eval set answers “did it actually help?” in seconds. Skip this step and you are tuning blind — every “improvement” is a vibe, not a measurement. If you also change how you cut documents, measure it with chunking strategies and Contextual Retrieval, because chunking moves these same metrics.

The cost and latency trade-off: what each stage adds in milliseconds

Every stage buys accuracy with latency, so weigh each one against your budget:

BM25 is nearly free — a keyword index lookup, single-digit milliseconds.
Dense retrieval is one embedding call for the query plus an ANN search — tens of milliseconds, dominated by the embedding round-trip.
RRF is pure arithmetic over a few hundred IDs — negligible.
Reranking is the expensive stage: a cross-encoder forward pass over ~100 (query, passage) pairs, plus a network round-trip if hosted. This is the biggest single latency add, which is exactly why you rerank 100 candidates and not 10,000.

BM25 keyword~single-digit ms, near-free

Dense + ANN~tens of ms, 1 embed call

RRF fusionnegligible, pure arithmetic

Cross-encoder reranklargest add; 1 call over ~100 pairs

When latency is tight, rerank-2.5-lite trades a little accuracy for lower latency and cost, and you can shrink the candidate pool. You can also cut generation cost independently with Claude prompt caching on the stable parts of your context. The discipline is the same throughout: turn each dial, re-run the eval set, and keep the setting only if the numbers earned it.

FAQ: hybrid vs rerank, RRF k, how many eval queries, when to skip reranking

Hybrid search or reranking — which first? Both, and they fix different things. Hybrid raises recall (gets the right chunk into the candidate pool); reranking raises precision (pushes it to the top-10). Reranking cannot promote a chunk that hybrid never retrieved, so build hybrid first, then rerank on top.

Why k=60 in RRF? It is the common default. The k constant damps how much a single top rank contributes, so no one list dominates and noisy top positions do not skew the fusion. Treat 60 as a sane starting point and tune it against your eval set.

How many eval queries do I need? Start with 30 to 50 labeled (query, relevant_chunk) pairs drawn from real usage. That is enough to catch regressions and rank pipeline variants; grow it as you find failure cases.

When can I skip reranking? When your eval numbers already clear your bar with hybrid alone, or when latency is so tight the extra cross-encoder call is unaffordable. The answer is in your Recall@k and MRR table, not in a rule of thumb — which is the entire point of measuring before and after.

For the full context on grounding the generated answer, see AI chatbot without hallucinations, and for how retrieval fits the rest of the stack, the ai-agents hub collects the companion guides — including the chunking strategies and Contextual Retrieval post that pairs with this one.

Fix a RAG That Retrieves the Wrong Chunks: Hybrid Search + Reranking (with Before/After Metrics, 2026)