Hybrid Search vs Vector Search: Why RAG Retrieval Needs Both

If you’re choosing between vector search and keyword search for a RAG system, the honest answer is that you want both, fused together. Vector search is strong on meaning and weak on exact identifiers; keyword search is the reverse. Real queries need both at once, and hybrid retrieval is the engineering response to that reality. This guide is how I actually build it — fusion, reranking, a concrete Postgres setup, and the evaluation loop most teams skip.

The one-line answer

Use hybrid. Run lexical (keyword/BM25) and semantic (vector) retrieval in parallel, fuse the two rankings, and rerank the top candidates. Pure vector search demos beautifully and then quietly underperforms in production the first time someone searches for an error code, a product SKU, or a person’s name. The reason isn’t a bad embedding model — it’s that similarity and exact-match are different jobs.

Why vector search alone quietly fails

Vector search embeds the query and the documents into the same space and returns nearest neighbors. That’s genuinely powerful: it matches “my laptop screen went black” to a document about “blank display troubleshooting” even though they share no keywords. For paraphrase and intent, nothing beats it.

But it has a failure mode that’s easy to miss in a demo and expensive in production: it’s fuzzy on exact tokens. Ask for account AC-90887 and an embedding will happily return something similar to that string — a different account number, a nearby ID — because in embedding space “close” is not “equal.” Names, error codes, version numbers, hashes, SKUs: these are exactly the tokens that a lot of real queries hinge on, and they’re exactly where pure vector retrieval is least reliable.

The trap is that vector-only search looks great on the curated questions you test with and falls down on the messy, identifier-heavy queries users actually type. By then it’s in production and the fix is a re-architecture, not a tweak.

Why keyword search alone fails

Keyword search (BM25 and friends) is the opposite. It’s precise: if the token is in the document, it finds it, and it ranks by how distinctive and frequent the match is. For identifiers it’s perfect.

And it’s brittle on meaning. Search “blank display” and a document that only ever says “black screen” scores zero, because lexical search matches strings, not concepts. Every paraphrase, synonym, and reformulation is a miss. On its own, keyword search makes users phrase things exactly the way the corpus does, which they never do.

So neither wins alone. One is precise and literal; the other is fuzzy and semantic. Most real queries are a mix — semantic intent plus a specific identifier — and that mix is why hybrid exists.

How hybrid actually works: fusion

Hybrid search runs both retrievers and combines their results. The combination step — fusion — is the part that’s an actual engineering decision, not an accident of whichever index you happened to query.

The boring, strong default is Reciprocal Rank Fusion (RRF). Instead of trying to reconcile two incompatible score scales (BM25 scores and cosine similarities don’t live on the same axis), RRF only uses each result’s rank in its own list:

def reciprocal_rank_fusion(result_lists, k=60):
    # result_lists: list of ranked lists of doc_ids, best-first
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

fused = reciprocal_rank_fusion([bm25_hits, vector_hits])

k is a smoothing constant (60 is a fine starting point). A document ranked highly by either retriever floats up; a document ranked highly by both floats higher. It’s robust precisely because it ignores the raw scores.

When you want more control, use a weighted blend instead — normalize each retriever’s scores and combine them with a weight you can tune per query type. That’s the point: an exact-identifier query should let the lexical signal dominate, while a “find me things like this” query should lean semantic. I keep the fusion weighting configurable and observable, because the right blend for narrative search is not the right blend for looking up a wallet address, and a single fixed weighting quietly serves one of them badly.

Reranking: the step teams skip

Fusion gets you a good candidate set. Reranking is what makes the top few results actually right, and it’s the single highest-leverage quality improvement in most stacks.

A cross-encoder reranker takes the query and each candidate document together and scores their relevance directly, rather than comparing two independently-computed vectors. That joint scoring catches relevance that neither first-stage retriever could see. The cost is latency — cross-encoders are heavy — so you never run them over the whole corpus. You retrieve broadly and cheaply with hybrid fusion, then rerank only the top N candidates:

candidates = fused[:50]                     # cheap, broad recall
reranked = cross_encoder.rank(query, candidates)   # expensive, precise
top_k = reranked[:8]                        # what the LLM actually sees

That two-stage shape — cheap-and-broad, then precise-and-bounded — is what lets the system be fast and genuinely relevant at the same time. Teams skip reranking because the first-stage results already “look fine,” and then wonder why answer quality plateaus. Most of the perceived “search quality” in a good RAG system is coming from this step.

A concrete setup: pgvector + BM25 in Postgres

You don’t need a dedicated vector database to start. If your data already lives in Postgres, you can do hybrid retrieval with pgvector for the semantic side and Postgres full-text search for the lexical side:

-- one row per chunk: the text, its full-text vector, and its embedding
CREATE TABLE chunks (
  id         bigserial PRIMARY KEY,
  document_id bigint NOT NULL,
  content    text NOT NULL,
  fts        tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
  embedding  vector(1024)          -- match your embedding model's dimensions
);

CREATE INDEX ON chunks USING gin (fts);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Then retrieve both ways and fuse. A compact version that does RRF right in SQL:

WITH lexical AS (
  SELECT id, row_number() OVER (
           ORDER BY ts_rank_cd(fts, plainto_tsquery('english', $1)) DESC) AS rnk
  FROM chunks
  WHERE fts @@ plainto_tsquery('english', $1)
  LIMIT 50
),
semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $2) AS rnk
  FROM chunks
  ORDER BY embedding <=> $2
  LIMIT 50
)
SELECT id, sum(1.0 / (60 + rnk)) AS score
FROM (SELECT * FROM lexical UNION ALL SELECT * FROM semantic) fused
GROUP BY id
ORDER BY score DESC
LIMIT 20;

$1 is the raw query text; $2 is the query embedding. This gives you real hybrid retrieval on infrastructure you already run, and you hand the top 20 to a reranker before the LLM sees them. Start here; reach for a dedicated vector store when scale or feature needs actually demand it, not before.

Chunking and metadata decide more than you think

A lot of retrieval quality is set before retrieval even runs, at indexing time. Chunk too coarsely and a precise answer gets diluted by surrounding irrelevant text; chunk too finely and you shred the context that made a passage meaningful. The right granularity depends on your documents and queries — treat it as a tuning decision, not a default you inherited from a tutorial.

Attach structured metadata to every chunk, too — source, entity, date, document type. It lets you filter and boost in ways pure similarity never can, which is decisive for queries shaped like “about this entity, from this source, in this window.” Teams spend weeks tuning fusion weights while leaving a naive chunking scheme in place, then wonder why quality is stuck. The retrieval layer can only rank what indexing gave it.

Evaluate, or you’re guessing

A hybrid stack is only as good as the evaluation behind it. You need a set of representative queries with judged results, so that when you change fusion weights or swap a reranker you can measure whether relevance went up instead of guessing. Without that, tuning is vibes — and vibes regress silently the first time the query mix shifts.

Build the eval loop in from the start: representative queries, judged relevance, and observability on where results are weak. That’s the difference between a search system you improve on purpose and one you shipped once and hope still holds.

When you don’t need hybrid

To be fair to the simpler options: if your corpus is small, your queries are all natural-language questions with no identifiers, and latency is critical, pure vector search may be enough — and skipping fusion and reranking is a legitimate simplification. Likewise, if users only ever search exact codes, plain full-text search is fine. Hybrid earns its complexity when your queries mix meaning and identifiers, which — for most real products — they do.

FAQ

Is hybrid search always better than vector search? Not universally, but for most real workloads, yes. Hybrid keeps vector search’s semantic recall while adding the exact-identifier precision that pure vector search lacks. The exception is a small corpus with purely natural-language queries and no identifiers, where vector-only can be enough.

What’s the simplest way to combine keyword and vector results? Reciprocal Rank Fusion. It combines the two ranked lists using only each result’s rank, so you don’t have to reconcile incompatible score scales. Start there; move to a tuned weighted blend when you need per-query-type control.

Do I need a dedicated vector database? Not to start. pgvector plus Postgres full-text search does real hybrid retrieval on infrastructure you already run. Move to a dedicated store when scale or specific features genuinely require it — not by default.

Where does reranking fit? After fusion, on a bounded candidate set. A cross-encoder reranker scores the query and each candidate together and fixes relevance the first-stage retrievers can’t see. It’s usually the highest-leverage quality step, and you keep it fast by only reranking the top N.

I design and audit retrieval systems like this for teams whose RAG works in the demo and disappoints in production. If that’s where you are — retrieval returning the wrong things, quality stuck, no eval loop — here’s how I work with people, or read the deeper hybrid search field notes.