Distributed Hybrid Search

"The vector index swore the answer was about meaning. The inverted index swore it was about the exact spelling of a product code. They were both right, which is why I now run both and let a referee settle it."

A Query Planner With Two Opinions

By this point in the chapter you can build a sharded approximate-nearest-neighbor index (Section 25.4) and replicate it for throughput and availability (Section 25.5). That index answers "what is semantically close to this query?" extremely well, and for a great deal of retrieval-augmented generation it is enough. The trouble starts at the edges of meaning. A support agent searches for the literal string E2401 and wants the one runbook that mentions that code; an embedding model, having never seen that token often enough to place it meaningfully, returns five passages that are vaguely about errors and none of them the right one. The same model, asked "how does the system grasp a sentence's gist?", confidently retrieves a passage that never uses the words "grasp", "sentence", or "gist", which is exactly what you wanted and exactly what a keyword index cannot do. Neither system is broken; they are tuned for different questions, and real query streams contain both.

1. Two Retrievers, Opposite Blind Spots Beginner

Dense retrieval encodes the query and every document into the same vector space and ranks by geometric closeness, so it scores documents on what they mean. Its strength is generalization: a query and a passage can rank highly together while sharing not a single word, because the embedding placed them near each other. That same generalization is its weakness for exact terms. A rare token (a stock-keeping unit, a hexadecimal hash, an unusual surname) appears too seldom in the training data for the model to give it a distinctive, reliable position, so it gets averaged into a fog of nearby tokens. Ask for that token literally and the dense index returns plausible neighbors instead of the exact hit.

Sparse lexical retrieval is the mirror image. An inverted index maps each term to the list of documents containing it (the postings list of Chapter 6), and a scoring function like BM25 ranks matches by how often the query terms appear, weighted by how rare each term is across the corpus. Exact tokens are its home turf: a unique code matches exactly one document and shoots to the top. But the inverted index has no notion of meaning. Two passages that say the same thing with different words share no terms, so a paraphrased query scores zero against the document the user actually wanted. The blind spots are complementary, which is the entire argument for running both.

2. Learned Sparse Retrieval: A Middle Ground Intermediate

Between the pure-meaning dense index and the pure-lexical inverted index sits a third option that has matured into production over the last few years: learned sparse retrieval. The idea is to keep the data structure of the inverted index (terms mapped to postings, with all its decades of distributed-serving engineering) but to let a transformer decide the term weights and, crucially, expand each document with related terms it does not literally contain. A model in the SPLADE family encodes a passage about "automobiles" and writes nonzero weights for "car" and "vehicle" into the sparse vector, so a lexical match on "car" now retrieves the "automobile" passage. The output is still a sparse vector over the vocabulary, indexable by the same sharded inverted-index infrastructure, but it carries some of the semantic generalization that used to require dense embeddings.

Learned sparse retrieval does not eliminate the need for fusion. It narrows the gap, often pushing a single index to recall that previously needed dense plus sparse, but the strongest systems still fuse a learned-sparse signal with a dense one because the two encoders make different mistakes. For the distributed-systems story the important point is that learned sparse retrieval is operationally a sparse index: it shards, replicates, and serves like the inverted index of Chapter 6, not like the ANN index of Section 25.4. That keeps it cheap to scale out, which is a large part of why practitioners reach for it.

3. Fusing Two Ranked Lists Intermediate

Once both indexes have each returned a ranked list of document identifiers, you must merge them into one. The obvious approach, adding the two relevance scores, runs straight into a problem: the scores are not comparable. A cosine similarity from the dense index lives in $[-1, 1]$; a BM25 score is an unbounded positive number whose scale depends on document length and corpus statistics. Summing them lets whichever scoring system happens to produce larger numbers dominate, regardless of which retriever is actually more confident. Any score-based fusion must therefore first normalize the two score distributions onto a common scale, and that normalization is fragile because the distributions shift with every query and every corpus update.

Reciprocal Rank Fusion (RRF) sidesteps the whole issue by throwing away the raw scores and keeping only the ranks. Each document gets a fused score that is the sum, over every list it appears in, of a term that decays with its rank in that list:

where $\text{rank}_i(d)$ is the position (starting at 1) of document $d$ in the $i$-th ranked list, $L$ is the number of lists being fused (two here), and $k$ is a small constant, conventionally 60, that damps the influence of very high ranks so the top one or two positions do not completely swamp the rest. A document ranked first in either list contributes $1/(k+1)$; a document that appears near the top of both lists accumulates two such terms and rises above documents that only one retriever liked. Because ranks are dimensionless, RRF needs no normalization and no per-corpus tuning, which is why it is the default fusion in most hybrid systems. Weighted score combination, $\alpha \cdot s_\text{dense} + (1-\alpha) \cdot s_\text{sparse}$ after normalizing each $s$, remains useful when you have labeled data to tune $\alpha$ and want to express that one retriever is generally more trustworthy, but it pays for that expressiveness with the normalization fragility RRF avoids.

The code below builds both retrievers from scratch over a tiny corpus, a hashless concept-based stand-in for a dense embedding and a real BM25 over an inverted index, then fuses them with RRF and measures recall on a query mix that deliberately contains both paraphrases and exact tokens.

import math, re
from collections import Counter

corpus = {
    1: "the model weighs every token against every other to capture long range structure",
    2: "engineers store learned vectors and serve approximate nearest neighbor lookups",
    3: "fault E2401 signals a stalled collective on the gradient interconnect",
    4: "the ranking function scores passages by how rare and frequent each word is",
    5: "a system understands the gist of a sentence rather than matching letters",
    6: "the obscure compound Zylophorb is named in exactly one forgotten note",
    7: "balancing the keyword query load means splitting the posting lists across hosts",
    8: "deep networks place related ideas near one another in a continuous space",
}

# A hand-built "semantic" model: meaning-bearing words map onto concept axes. It
# generalizes across paraphrases but, like a real embedding, blurs rare tokens
# (codes, names) by simply not placing them on any axis.
CONCEPTS = {
    "attention": ["model","weighs","token","capture","structure","attention","context",
                  "long","range","distant","words","spans","attends"],
    "vectors":   ["store","vectors","learned","embeddings","nearest","neighbor","approximate",
                  "lookups","space","continuous","near","nearby","related","ideas","place","placed"],
    "semantics": ["understands","gist","sentence","meaning","semantic","semantics","rather",
                  "matching","letters","deep","networks","understand","grasp"],
    "ranking":   ["ranking","scores","passages","rare","frequent","word","function"],
    "sharding":  ["balancing","keyword","query","load","splitting","posting","lists","hosts"],
}
AXIS = {w: i for i, ws in enumerate(CONCEPTS.values()) for w in ws}
DIM = len(CONCEPTS)
tokenize = lambda t: re.findall(r"[a-z0-9]+", t.lower())

def embed(text):
    v = [0.0] * DIM
    for tok in tokenize(text):
        if tok in AXIS:                 # codes / rare names land on no axis: blurred away
            v[AXIS[tok]] += 1.0
    n = math.sqrt(sum(x*x for x in v)) or 1.0
    return [x / n for x in v]

doc_vecs = {d: embed(t) for d, t in corpus.items()}
cosine = lambda a, b: sum(x*y for x, y in zip(a, b))

def dense_search(query, k=5):
    q = embed(query)
    return sorted(corpus, key=lambda d: cosine(q, doc_vecs[d]), reverse=True)[:k]

# BM25 over an inverted-index-style term statistic (the Chapter 6 posting lists).
N = len(corpus)
doc_tokens = {d: tokenize(t) for d, t in corpus.items()}
doc_len = {d: len(toks) for d, toks in doc_tokens.items()}
avgdl = sum(doc_len.values()) / N
df = Counter(tok for toks in doc_tokens.values() for tok in set(toks))
idf = lambda term: math.log(1 + (N - df.get(term, 0) + 0.5) / (df.get(term, 0) + 0.5))

def bm25_search(query, k=5, k1=1.5, b=0.75):
    scores = {}
    for d in corpus:
        tf = Counter(doc_tokens[d]); s = 0.0
        for term in tokenize(query):
            if term in tf:
                f = tf[term]
                s += idf(term) * (f*(k1+1)) / (f + k1*(1 - b + b*doc_len[d]/avgdl))
        if s > 0:
            scores[d] = s
    return sorted(scores, key=scores.get, reverse=True)[:k]

def rrf(rank_lists, k=60):                          # reciprocal rank fusion
    fused = {}
    for ranked in rank_lists:
        for rank, doc in enumerate(ranked):
            fused[doc] = fused.get(doc, 0.0) + 1.0 / (k + rank + 1)
    return sorted(fused, key=fused.get, reverse=True)

def hybrid_search(query, k=5):
    return rrf([dense_search(query, k), bm25_search(query, k)])[:k]

queries = [                                         # mix of paraphrase and exact-term needs
    ("attention spans distant context", 1),         # paraphrase: zero word overlap with doc 1
    ("grasp semantics not letters", 5),             # paraphrase: zero word overlap with doc 5
    ("related ideas placed nearby", 8),             # paraphrase
    ("E2401", 3),                                   # exact code, dense blurs it away
    ("Zylophorb", 6),                               # rare exact token, dense blurs it away
    ("posting lists keyword load", 7),              # exact keywords, sparse nails it
]
recall_at_k = lambda fn, k=3: sum(g in fn(q, k) for q, g in queries) / len(queries)
print("dense  recall@3      :", f"{recall_at_k(dense_search):.2f}")
print("sparse recall@3      :", f"{recall_at_k(bm25_search):.2f}")
print("hybrid recall@3 (RRF):", f"{recall_at_k(hybrid_search):.2f}")

dense  recall@3      : 0.83
sparse recall@3      : 0.83
hybrid recall@3 (RRF): 1.00

Output 25.6.1 is the whole argument in three numbers. Dense and sparse each score $0.83$, but on different queries, so fusing their ranked lists with the parameter-free RRF of the formula above reaches $1.00$. Nothing in the fusion step looked at a raw score; it saw only ranks, which is what let it combine a cosine similarity and a BM25 value without ever putting them on the same scale.

4. Metadata Filtering: Pre-Filter or Post-Filter Intermediate

Real retrieval rarely asks only "what is relevant?"; it asks "what is relevant among the documents I am allowed to see, in this language, newer than last quarter?" Those constraints are metadata filters, and combining them with vector search forces a choice that has direct cost and recall consequences. Pre-filtering evaluates the metadata predicate first, then searches only the surviving subset. It guarantees that every returned result satisfies the filter and that you get a full top-$k$ of valid documents, but it fights the structure of an approximate-nearest-neighbor index: the graph or inverted-list traversal of Section 25.4 is built to walk the whole space, and confining it to a small allowed subset can wreck its efficiency or force a brute-force scan of the survivors. When the filter is highly selective (a single user's private documents), pre-filtering is usually right because the survivor set is small.

Post-filtering runs the ANN search over everything, retrieves a top-$k$, then discards results that fail the predicate. It keeps the index fast because the search is unconstrained, but it risks returning fewer than $k$ valid results, or none, when the filter is selective and all the nearest neighbors happen to be filtered out. The common mitigation is to over-fetch, retrieving the top $c \cdot k$ for some cushion $c$ and filtering down, but choosing $c$ is a guess that trades latency against the chance of an underfull result. The honest summary is a trade-off table, not a winner: pre-filter when the predicate is selective and post-filter when it is permissive, and many vector databases now implement filtered ANN that interleaves the two so the traversal skips disallowed nodes as it walks. The sharding from Section 25.5 complicates this further, since a filter that aligns with the shard key can prune entire shards before any search runs, while a filter orthogonal to the shard key touches every shard regardless.

5. The Latency Cost of Running Two Indexes Advanced

Hybrid search buys recall with infrastructure. You now operate two distributed retrieval systems, each sharded and replicated on its own terms, and every query touches both. The saving grace is that the two are independent, so a query fans out to the dense and sparse indexes concurrently and the hybrid latency is the maximum of the two, not their sum, plus a small fusion cost. That maximum, though, is governed by the slower index and by the tail behavior of its shards: a hybrid query is not done until both its scatter-gather rounds complete, so it inherits the worst straggler from either system, the same tail-latency dynamic that Section 25.5 raised for a single sharded index, now doubled. The fusion step itself is cheap, merging two short ranked lists of a few hundred candidates, but the two over-fetches that feed it (each index typically returns more than the final $k$ so fusion has material to work with) widen the candidate set that any downstream reranker must process.

There is also an operational tax that does not show up in a latency number. Two indexes mean two ingestion pipelines, two sharding schemes that can drift out of alignment, and two failure domains; a document added to the vector index but not yet to the inverted index is invisible to half of every hybrid query until both catch up. Keeping the two indexes consistent under a stream of updates is its own distributed-systems problem, a cousin of the dual-write consistency issues that haunt any system maintaining two copies of derived data. The payoff has to clear this combined bar, which is why hybrid search is the default for production RAG over heterogeneous corpora but overkill for a homogeneous, paraphrase-light corpus where one index already answers nearly everything.

# Qdrant: dense + sparse fan-out fused by RRF, server-side and sharded.
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
hits = client.query_points(
    collection_name="docs",
    prefetch=[
        models.Prefetch(query=dense_vec, using="dense", limit=50),    # ANN branch
        models.Prefetch(query=sparse_vec, using="sparse", limit=50),  # learned-sparse branch
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),               # the merge from Section 3
    limit=10,
).points

With both retrievers fused, the candidate list is broad and recall is high, but the very top of that list is not yet ordered with the precision a generator needs. Hybrid search optimizes for getting the right document into the candidate set; putting the single best document at rank 1 is a separate, more expensive computation. That is the job of reranking, and stacking a distributed reranker on top of the fused list is where Section 25.7 takes the story next.