Reranking in RAG is a second retrieval pass: you take the top candidates from your first-stage search and re-score them with a model that reads the query and each document together, then keep only the best few for the LLM. The model that does this is almost always a cross-encoder, and adding one is usually the single highest-leverage change you can make to a RAG system that retrieves plausible-looking chunks but still answers wrong. This guide is how I actually add a reranker — the mechanics, which model to pick, how to size the candidate set, the latency budget, and how to prove it helped instead of assuming it did.
The one-line answer
Retrieve broadly and cheaply, then rerank narrowly and precisely. Your first stage (vector, BM25, or hybrid fusion) pulls 50–100 candidates optimizing for recall — get the right document in the set somewhere. The reranker then re-orders those candidates for precision and you pass the top 5–10 to the LLM. The reason this two-stage shape wins is that the accurate scoring model is too slow to run over your whole corpus, and the fast retrieval model is too imprecise to trust for the final ordering. Reranking is how you get both.
If you already run hybrid retrieval, reranking is the step after fusion. If you’re still on pure vector search, a reranker is often a bigger quality jump than switching embedding models — and much cheaper to try.
Why the second pass sees what the first can’t
The whole point of a reranker rests on one architectural distinction: bi-encoders versus cross-encoders.
Your first-stage retriever uses a bi-encoder. It embeds the query into a vector and — separately, usually at indexing time — embeds every document into a vector. Retrieval is then just nearest-neighbor math over those precomputed vectors. This is what makes vector search fast: the documents were encoded once, offline, and the query never actually “meets” the document. The model compresses each document into a fixed vector before it knows what you’ll ask, so any nuance that only matters for a specific query has already been flattened away.
A cross-encoder does the opposite. It takes the query and one candidate document, concatenates them, and runs them through a transformer together, with full attention across both texts. The output is a single relevance score. Because the model reads the pair jointly, it can catch things the bi-encoder structurally cannot: that the document mentions your exact entity but in a negated clause, that two passages look similar in embedding space but only one actually answers the question, that a keyword match is coincidental rather than topical.
The tradeoff is symmetrical to the strength. You cannot precompute cross-encoder scores, because the score doesn’t exist until the query arrives. So you must run the model live, once per candidate, at query time. That’s why you never point a cross-encoder at your whole index — you point it at the small candidate set the fast retriever already narrowed down.
That asymmetry — bi-encoder for cheap-and-broad recall, cross-encoder for expensive-and-precise ordering — is the entire design.
The two-stage shape and how many candidates to rerank
The two numbers that matter are how many candidates you retrieve (the rerank depth) and how many you keep (the top-k the LLM sees).
- Retrieve depth (N): typically 50–100. This is a recall dial. If the right document isn’t in the candidate set, no reranker can save you — reranking only re-orders what it’s given. Too shallow and you miss answers; too deep and you pay latency reranking junk. I start at 50 and tune with evaluation, not vibes.
- Keep (top-k): typically 5–10. This is what actually reaches the LLM’s context. Reranking’s job is to make sure the right handful is at the top of a set that recall made broad.
The mental model: recall is the first stage’s job, precision is the reranker’s job. You deliberately over-retrieve so the reranker has room to work, then you deliberately cut hard so the LLM isn’t drowning in marginal context. A common failure is retrieving 10 and reranking 10 — that gives the reranker almost nothing to fix, and you’ve paid for a second model to shuffle a list that was already fixed by the first stage.
A concrete implementation
Here’s a real, local reranker using a cross-encoder from sentence-transformers. No API, no vendor — this runs on your own box:
from sentence_transformers import CrossEncoder
# A strong open reranker. Runs on CPU; much faster on a GPU.
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
def rerank(query: str, candidates: list, top_k: int = 8):
# candidates: objects from your first-stage retriever, each with .text
pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs) # one relevance score per (query, doc) pair
ranked = sorted(zip(candidates, scores), key=lambda cs: cs[1], reverse=True)
return [c for c, _ in ranked[:top_k]]Wired into a hybrid pipeline, the flow is:
def retrieve(query, query_embedding):
lexical = bm25_search(query, limit=50) # exact-match recall
semantic = vector_search(query_embedding, limit=50) # semantic recall
fused = reciprocal_rank_fusion([lexical, semantic]) # combine ranks
candidates = fused[:50] # broad candidate set
return rerank(query, candidates, top_k=8) # precise final orderingTwo implementation notes that matter in production. First, predict runs the model in batches under the hood, but the number of pairs is your latency multiplier — reranking 50 candidates is roughly 50× the work of scoring one, so the retrieve depth is a latency decision, not just a quality one. Second, max_length truncates long documents; if your chunks are large, the reranker only sees the first ~512 tokens of each, which is another reason chunking decisions upstream quietly cap what reranking can do.
Picking a reranker: open, hosted, or LLM
There are three families, and the right one depends on your latency budget, data-sensitivity, and how much infra you want to run.
Open cross-encoders you host yourself — the bge-reranker family, mxbai-rerank, and similar. Strong quality, no data leaves your environment, no per-query cost. The price is that you run the inference: a cross-encoder is a real model, and reranking 50 candidates on CPU can cost hundreds of milliseconds. On a GPU it’s fast; on CPU you’ll feel it. This is my default when data can’t leave the building or query volume makes per-call pricing painful.
Hosted reranking APIs — Cohere Rerank, Jina, Voyage. You POST the query and candidates, you get scores back. Zero infra, consistently strong models, and someone else owns the GPUs. The costs are a network round-trip on your critical path, per-query pricing that adds up at scale, and sending your query and retrieved documents to a third party — which is a real constraint for regulated or sensitive corpora. Great when you want quality fast and don’t want to run models.
LLM-as-reranker — prompt a general LLM to score or order the candidates. Maximally flexible and can reason about relevance in ways a dedicated reranker can’t, but it’s the slowest and most expensive option, and its ordering is less stable. I reach for this only for low-volume, high-value queries where the reasoning genuinely helps, not as a default retrieval component.
For most teams starting out, the honest recommendation is: try a hosted API first to confirm reranking helps at all on your data (it’s a one-hour experiment), then decide whether to bring a bge-reranker in-house for cost or privacy once you’ve proven the lift.
The latency budget is the real constraint
Reranking’s quality story is easy. The engineering story is the latency budget, and it’s where reranking projects actually get decided.
A cross-encoder pass adds real time — commonly tens to a few hundred milliseconds depending on model size, candidate count, and hardware. That has to fit inside whatever end-to-end budget your product allows, alongside first-stage retrieval and the far larger LLM generation call. The levers when it doesn’t fit:
- Reduce retrieve depth. Reranking 30 instead of 60 roughly halves reranker latency. Use evaluation to find the depth where recall stops improving — reranking candidates the first stage ranked 80th is usually wasted work.
- Use a smaller reranker. The
bge-rerankerfamily and others ship in multiple sizes; a smaller model gives back most of the quality for a fraction of the latency. - Run it on the right hardware. Cross-encoders are GPU-friendly. If you’re reranking on CPU and it hurts, a small GPU or a hosted API often solves the latency problem outright.
- Rerank in parallel with nothing. It sits on the critical path between retrieval and generation — you can’t hide it behind other work, so the budget has to genuinely accommodate it.
The point is to treat the reranker as a component with an SLA, not a free accuracy boost. If it blows your latency budget, a slightly-worse-but-fast reranker beats a perfect one that makes the product feel broken.
Failure modes I actually see
Reranking a set the first stage already ruined. The reranker only re-orders its input. If first-stage recall is bad — the right document isn’t in the top 50 — reranking polishes a set that never contained the answer. Fix recall first; a reranker is a precision tool, not a recall tool.
Chunks longer than the reranker’s context. If your chunks exceed the reranker’s max_length, it scores only the truncated head of each document and silently misjudges the rest. Reranking and chunking are coupled; you can’t tune one blind to the other.
Trusting the vendor’s headline number. Blog posts love “+40% accuracy.” That’s their dataset, not yours. Reranking’s lift on your corpus and your query mix might be dramatic or marginal — the only way to know is to measure it, which is the next section.
Reranking when the first stage was already right. On easy, unambiguous queries the top result is often correct before reranking. The reranker’s value shows up on the hard, ambiguous, identifier-plus-intent queries — which is exactly why an averaged metric can hide a big win on the queries that matter and a wash on the ones that didn’t need help.
Prove it worked, don’t assume it
Adding a reranker feels like it should help, and that feeling is how teams ship a slower pipeline that isn’t actually better. You need an evaluation loop, and it doesn’t have to be fancy.
Build a set of representative queries with judged relevant documents — even 50 hand-labeled queries beat zero. Then measure retrieval quality with and without the reranker on the same set, using a ranking metric like NDCG@10 or a simple hit-rate at your real top-k. Now the question “did reranking help?” has an answer instead of an opinion, and you can compare rerankers, tune retrieve depth, and catch the day a model swap silently regresses relevance.
Two things to watch. First, evaluate at the top-k the LLM actually receives — improving rank 40 is irrelevant if you only pass 8. Second, look at per-query deltas, not just the average: reranking often helps the hard queries a lot and the easy ones not at all, and the average flattens exactly the signal you care about. I keep this loop from day one, because a reranker you can’t measure is a reranker you’re tuning by superstition.
When you don’t need a reranker
Reranking earns its latency; it isn’t free, so be honest about when to skip it. If your corpus is small and your first-stage retrieval already puts the right answer at the top on your eval set, a reranker adds latency for a lift you can’t measure — don’t add it. If you’re on a hard real-time budget where even 50ms is unacceptable, the two-stage shape may not fit, and you’re better off investing in first-stage quality. And if you haven’t built any evaluation yet, add that before the reranker — otherwise you’re stacking an unmeasured component on an unmeasured baseline and calling it an improvement.
For most production RAG systems, though, where queries are messy and answer quality has plateaued, a reranker is the highest-leverage next step. It’s a smaller change than re-architecting retrieval and usually a bigger payoff than another round of prompt-tuning.
FAQ
Is a reranker the same as a cross-encoder? In practice, almost always yes. “Reranker” describes the job (re-order the candidate set for precision); “cross-encoder” describes the architecture that does it well (reads query and document together and scores the pair). There are LLM-based and other rerankers, but a cross-encoder is the standard, and when people say “add a reranker” they usually mean add a cross-encoder.
How many documents should I rerank? Start by retrieving ~50 candidates and passing the top 5–10 to the LLM. Retrieve depth is a recall-vs-latency dial and top-k is a context-quality dial; tune both against an eval set rather than inheriting numbers from a tutorial.
Does reranking replace hybrid search? No — they’re complementary and they stack. Hybrid fusion improves the candidate set (recall); reranking improves the ordering of that set (precision). The strongest setups do both: hybrid retrieval to build a broad candidate set, then a cross-encoder to order it.
Will a reranker fix bad retrieval? Only the precision half of it. A reranker can only re-order documents it’s given, so if first-stage recall misses the answer entirely, reranking can’t recover it. Fix recall first (better chunking, hybrid retrieval), then add reranking for precision.
Hosted API or self-hosted model?
Start hosted (Cohere/Jina/Voyage) to confirm reranking helps on your data with an hour of work. Move to a self-hosted open cross-encoder like bge-reranker when per-query cost or data-privacy constraints justify running the inference yourself.
I design and audit retrieval systems for teams whose RAG works in the demo and disappoints in production — retrieval returning plausible-but-wrong chunks, answer quality stuck, no way to tell whether a change helped. If that’s where you are, this is the hybrid search & RAG guide hub, the way I think about relevance scoring as a capability, and how I work with people. For the layer beneath reranking, start with how to build hybrid search with pgvector and BM25.