Why Is My RAG Retrieval Bad? A Diagnostic Order of Operations

Your RAG retrieval is bad for one of a small number of specific reasons, and they need opposite fixes — which is why swapping the embedding model, raising top_k, and rewriting the prompt in the same afternoon usually leaves you exactly where you started. The answer is either not in the corpus, not in the index, split across chunk boundaries, present but ranked below your cutoff, filtered out by metadata, or retrieved correctly and then ignored by the LLM. This guide is the diagnostic order I actually run: how to localize the failure to one of those stages first, and only then reach for the fix that matches it.

The one-line answer

Bad retrieval is a localization problem before it’s a tuning problem. Every RAG pipeline is a chain — corpus, index, chunk, embed, rank, cut, generate — and a bad answer means the correct evidence was lost at exactly one of those links. Find the link that dropped it, and the fix is usually obvious and small. Skip that step and you’ll spend a month tuning chunk sizes to fix a document that was never ingested.

The most expensive habit I see is teams treating “retrieval is bad” as a single symptom with a single cure. It isn’t. It’s a stack of failures that all look identical from the chat window, and the whole job is telling them apart.

Step 0: look at the chunks, not the answer

Before any theory: take a query that produces a wrong answer and print what was actually retrieved. Every chunk, its score, its source document, its rank. Nobody wants to do this and it resolves a startling share of cases in under a minute.

You’re asking one question: is the correct evidence in the retrieved set at all?

• The answer text is in the retrieved chunks, and the LLM still got it wrong. Retrieval is fine. Stop tuning retrieval. Your problem is downstream — context ordering, prompt, chunk truncation, or a model that ignored evidence in favour of its own priors. Jump to the generation section.
• The answer text isn’t there. Retrieval genuinely failed, and now you get to find out where. Keep reading.

That branch matters because those two worlds share zero fixes. I’ve watched a team spend two sprints on chunking strategy for a system whose retrieved chunks contained the right answer on almost every failing query. The retriever was doing its job. The prompt was burying the evidence under three thousand tokens of boilerplate instructions.

If you can’t print retrieved chunks with scores today, build that before anything else. A pipeline you can’t inspect is a pipeline you’ll tune by superstition.

The recall ceiling test

When the evidence isn’t in the retrieved set, the next question is not “which knob do I turn” — it’s could this system have found it at all?

Pick a handful of failing queries. For each one, find by hand the document (and the specific passage) that actually answers it. Then run this check:

def diagnose(query: str, gold_doc_id: str, retrieve, top_k: int = 8):
    """Localize where a known-correct document falls out of the pipeline."""
    # Retrieve much deeper than production so we can see the whole ranking.
    candidates = retrieve(query, limit=200)
    ids = [c.doc_id for c in candidates]

    if gold_doc_id not in ids:
        return "NOT_RETRIEVABLE"   # not in index, filtered out, or embedding mismatch

    rank = ids.index(gold_doc_id) + 1
    if rank > top_k:
        return f"RANKED_TOO_LOW (rank {rank}, cutoff {top_k})"

    return f"RETRIEVED (rank {rank}) — failure is downstream"

Three outcomes, three completely different investigations:

NOT_RETRIEVABLE — the document isn’t reachable by this query at any depth. This is an ingestion, chunking, filtering, or representation problem. It is not a ranking problem, and no amount of reranking or top_k tuning will touch it.

RANKED_TOO_LOW — the system found it and buried it. This is a ranking problem, and it’s the good news case: reranking and better fusion are built for exactly this.

RETRIEVED — retrieval already worked at your production cutoff. The loss is downstream.

That single check collapses “why is my RAG retrieval bad” from an open-ended question into one of three narrow ones. Everything below is what to do with each answer.

Failure 1: the document was never really in the index

Boring, common, and the first thing I check when a query returns NOT_RETRIEVABLE. Query your index directly, by document id or a distinctive exact phrase, with no vector search involved. If it isn’t there, retrieval was never the problem.

What actually causes this in production:

• Ingestion silently dropped it. A parser failed on one PDF in a batch of four thousand and the error went to a log nobody reads. Scanned pages that needed OCR came through as empty strings. Tables and multi-column layouts turned into scrambled text that no longer contains the sentence you’re looking for.
• The index is stale. The document was added after the last embedding run, or an update rewrote the source and the vector still encodes the old version. Every RAG system I’ve audited that had “occasional missing answers” had some version of this.
• The embedding call failed and was swallowed. Rate limits, timeouts, a batch that raised halfway through — and the row exists with a null or zeroed vector, so it never matches anything.

The fix is not clever. Count documents in, count chunks and vectors out, reconcile the two, and alert when they diverge. Ingestion needs the same reliability posture as any other pipeline: explicit failure handling, retries that don’t silently give up, and a dead-letter path for documents that couldn’t be parsed. This is the same discipline I describe in designing systems that survive disruption — an ingest path that fails quietly is worse than one that fails loudly, because it degrades answer quality invisibly and slowly.

Failure 2: chunking destroyed the answer

If the document is indexed but still unreachable, look at the actual chunk text. Not the strategy, not the config — the strings.

The two failure shapes:

The answer got split. The question needs a fact and its qualifier — a value in one sentence, the condition it applies under in the next — and the chunk boundary landed between them. Each half is now individually unconvincing, so neither ranks, and if one does retrieve, it’s misleading rather than merely useless.

The answer got diluted. The chunk is 2,000 tokens of mostly-unrelated text with one relevant sentence in the middle. The embedding is an average of everything in it, so that one sentence barely moves the vector. It exists in the index and is functionally invisible.

Both are the same underlying mistake: chunking on character counts rather than on the document’s own structure. Split on real boundaries — headings, sections, list items, table rows — and let chunk length vary. Add modest overlap so a fact and its qualifier are unlikely to be separated in every chunk. For structured documents, prepend the section path to the chunk text so an isolated paragraph still carries the context that identifies it.

One asymmetry worth knowing: the chunk you retrieve on and the chunk you give the LLM don’t have to be the same. Embed small, focused units for retrieval precision, then expand to the surrounding section before generation. That decoupling fixes a lot of “the right chunk retrieved but there wasn’t enough context to answer” cases without making your embeddings mushier.

Failure 3: the query and the document don’t live in the same neighbourhood

Document indexed, chunk intact, still NOT_RETRIEVABLE. Now it’s a representation problem — the embedding of the query and the embedding of the chunk aren’t close, even though a human sees the match instantly.

Where pure vector search structurally struggles:

• Exact identifiers. Error codes, SKUs, ticket numbers, function names, version strings. Embeddings encode meaning, and ERR_5521 doesn’t have any. It’s a token that either matches or doesn’t, and vector similarity is the wrong instrument for it.
• Vocabulary mismatch. Users ask in their words; documents are written in the organization’s. “Why did my card get declined” versus a policy document that only ever says “authorization failure.” The concepts are related, but a general-purpose embedding model may not put them close enough to beat a few thousand competing chunks.
• Negation and near-duplicates. Two passages describe the same feature; one says it’s supported, the other says it was removed in v3. In embedding space they’re nearly identical, so which one you get is close to a coin flip.

The first two are the strongest argument for not running pure vector search in the first place — a lexical arm catches exact tokens that embeddings smear, and the fusion of both is more robust than either. That’s the case I make in hybrid search vs vector search, and the concrete build is in hybrid search with pgvector and BM25. If your NOT_RETRIEVABLE cases are dominated by identifiers and exact phrases, adding a lexical arm will move your recall further in a day than a month of embedding-model comparisons.

The third — near-duplicates that differ in one decisive clause — is what a cross-encoder is genuinely good at, because it reads the query and the document together instead of comparing two precomputed averages.

Failure 4: ranked, but below the cutoff

RANKED_TOO_LOW is the friendliest diagnosis, because the information is present and only the ordering is wrong.

Resist the obvious move. Raising top_k “fixes” it in the sense that the right chunk now appears — alongside twelve marginal ones, which is its own quality problem: more context is not better context, and models measurably lose evidence buried in the middle of a long context window. You’ve traded a retrieval failure for a generation failure and made both harder to see.

The structural fix is the two-stage shape: retrieve broadly for recall (50–100 candidates), then re-score narrowly for precision and pass a small top-k to the LLM. That’s what reranking with a cross-encoder is for, and RANKED_TOO_LOW is precisely the failure it addresses — the item was in the candidate set; it just wasn’t at the top.

If you’re already running hybrid retrieval and things rank strangely, the fusion itself is worth auditing before you add another model: which arm is contributing the winners, whether one arm dominates, whether your score normalization is doing something silly across queries with different score distributions. I go through that tuning process in tuning hybrid search.

Failure 5: metadata filters quietly removed it

An underrated cause of NOT_RETRIEVABLE, because the query looks fine, the document is indexed, and the chunk is perfect — and a WHERE clause deleted it before scoring.

Common versions: a tenant or permission filter that’s stricter than intended; a date filter defaulting to the last 90 days on a corpus whose important documents are older; a doc_type filter that excludes the one type that answers this class of question; a filter applied before vector search that leaves too few candidates for the ranking to be meaningful.

The test is trivial — run the query with all filters removed. If the answer appears, you’ve found it. Then decide whether the filter is wrong or the routing that chose it is. This one is easy to miss precisely because filters are usually correct; when they’re wrong they fail silently and completely.

Failure 6: retrieval was fine and generation lost it

Back to the RETRIEVED branch. The right chunk was in context and the answer was still wrong. Things I check, in order:

• Position. Where did the correct chunk land in the assembled prompt? Evidence in the middle of a long context gets used less reliably than evidence at the edges. If you pass 20 chunks and the right one is number 11, that’s a plausible cause on its own.
• Contradiction. Did another retrieved chunk say something conflicting — an older policy version, a deprecated setting? The model isn’t wrong to be confused; two of your chunks disagree and nothing told it which one wins. Recency and version metadata belong in the chunk text, not just in the filter layer.
• Truncation. Is your context assembly silently cutting chunks to fit a token budget? Half a table is worse than no table.
• Instructions overriding evidence. A system prompt that says “answer concisely from your knowledge” will do exactly that, and the retrieved context becomes decoration.

Note that only the first two are properly RAG problems. The others are plumbing, and plumbing is where a surprising number of “the retrieval is bad” tickets actually terminate.

Build the eval loop, or you’re guessing forever

Everything above diagnoses one query at a time. That’s the right way to start, and the wrong way to operate — because the fix for one failure class routinely regresses another, and single-query debugging can’t see that. Shrink chunks to sharpen retrieval precision and you fracture the multi-sentence answers. Add a lexical arm for identifiers and some conceptual queries get worse. Without measurement you’re playing whack-a-mole with a blindfold on.

The minimum viable version is small:

def recall_at(queries, retrieve, n=50):
    """queries: [(query_text, gold_doc_id)] — 50 hand-labeled pairs is enough to start."""
    hits, misses = 0, []
    for q, gold in queries:
        ids = [c.doc_id for c in retrieve(q, limit=n)]
        if gold in ids:
            hits += 1
        else:
            misses.append(q)          # the miss list is the real deliverable
    return hits / len(queries), misses

Two things make this useful rather than ceremonial. First, measure recall at retrieve-depth and precision at the top-k the LLM actually sees — they’re different failures and one number hides both. Second, read the miss list, don’t just track the aggregate. The misses cluster, and the clusters are the diagnosis: all identifiers, all one document type, all questions spanning two sections. That clustering tells you which of the failures above you actually have, at population scale rather than anecdote scale.

Fifty labeled query–document pairs is a real afternoon of work and it is the highest-leverage afternoon in this entire guide. It converts “retrieval feels bad” into “recall@50 is 0.71 and every miss is an error code” — and the second sentence contains its own fix.

The order I’d actually work in

If you want the compressed version:

1. Print the retrieved chunks. Split retrieval failures from generation failures before touching anything.
2. Run the recall ceiling test on 5–10 known-bad queries. Get each one to NOT_RETRIEVABLE, RANKED_TOO_LOW, or RETRIEVED.
3. Drop the filters and re-run. Cheapest possible check, and it’s occasionally the whole answer.
4. Verify ingestion and index freshness for the specific documents that failed.
5. Read the actual chunk text for split and dilution damage.
6. Add a lexical arm if the misses are exact tokens, identifiers, or vocabulary mismatch.
7. Add reranking if things are retrieved but ranked below the cutoff.
8. Build the 50-query eval set so steps 6 and 7 are decisions instead of hopes.

Notice that changing the embedding model isn’t on the list. It’s the most common first move and it’s rarely the highest-value one: it’s expensive (full re-index), it changes everything at once, and it fixes only the representation failure — one of six. Do it when your eval set says representation is the bottleneck, not because a leaderboard moved.

When your retrieval isn’t actually the problem

Some honesty about when this whole exercise is misapplied.

If your corpus genuinely doesn’t contain the answer, no retrieval architecture will produce one. A surprising share of “bad RAG” complaints are questions whose answers live in a system nobody indexed, or exist only in someone’s head. The correct fix is content, not code — and a system that says “I don’t have that” is more valuable than one that confabulates.

If your corpus is small — a few hundred documents — you may not need retrieval sophistication at all. Long-context models can take a substantial document set directly, and the honest comparison is against that baseline, not against a worse RAG.

And if the questions are aggregate or analytical — “how many,” “which customers,” “trend over time” — retrieval is the wrong tool entirely. Those are queries against structured data, not similarity searches over prose. Semantic retrieval will return chunks that discuss the topic and the model will invent a number from them. That’s not a tuning failure; it’s an architecture mismatch, and the fix is a query layer, not a better index.

FAQ

Should I fix chunking or add reranking first? Whichever your diagnosis points at. If the correct document is NOT_RETRIEVABLE, reranking cannot help you — a reranker only re-orders what the first stage already found. If it’s retrieved but ranked low, reranking is the direct fix and chunking changes are a detour.

Will a better embedding model fix bad retrieval? Sometimes, and it’s rarely the first move. It only addresses the representation failure, it requires a full re-index, and it changes everything simultaneously so you can’t tell what improved. Prove representation is your bottleneck on an eval set before paying for it.

Is more context better? Should I just raise top_k? No. Beyond a modest number, extra chunks add distractors and push real evidence into the middle of the context window where models use it least reliably. Raising top_k masks a ranking problem instead of solving it; retrieve deep and cut narrow instead.

How do I know if it’s retrieval or the LLM? Print the retrieved chunks and read them. If the answer is in there and the output is still wrong, it’s generation. If it isn’t, it’s retrieval. This one check saves more time than any other in this guide.

How many labeled queries do I need before the numbers mean anything? Fifty is enough to steer with, and enormously better than zero. You’re not publishing a benchmark — you’re trying to tell whether a change helped, and to see which failures cluster.

My retrieval scores look high but the answers are wrong. What’s happening? Similarity scores measure closeness in embedding space, not correctness. A confidently-scored chunk about the right topic that doesn’t contain the answer is the classic distractor, and it’s exactly the case a cross-encoder is better at judging than a bi-encoder — because it reads the query and the document together instead of comparing two precomputed averages.

The retrieval layer this diagnosis sits on top of — hybrid fusion, reranking, chunking, and the evaluation loop that keeps them honest — is collected in my guides on hybrid search and RAG, and turning a retrieval stack that ranks plausible-but-wrong results into one you can measure and trust is what I do in relevance and correlation scoring work.