Distributed Hybrid Search

"The vector index swore the answer was about meaning. The inverted index swore it was about the exact spelling of a product code. They were both right, which is why I now run both and let a referee settle it."

A Query Planner With Two Opinions

By this point in the chapter you can build a sharded approximate-nearest-neighbor index (Section 25.4) and replicate it for throughput and availability (Section 25.5). That index answers "what is semantically close to this query?" extremely well, and for a great deal of retrieval-augmented generation it is enough. The trouble starts at the edges of meaning. A support agent searches for the literal string E2401 and wants the one runbook that mentions that code; an embedding model, having never seen that token often enough to place it meaningfully, returns five passages that are vaguely about errors and none of them the right one. The same model, asked "how does the system grasp a sentence's gist?", confidently retrieves a passage that never uses the words "grasp", "sentence", or "gist", which is exactly what you wanted and exactly what a keyword index cannot do. Neither system is broken; they are tuned for different questions, and real query streams contain both.

1. Two Retrievers, Opposite Blind Spots Beginner

Dense retrieval encodes the query and every document into the same vector space and ranks by geometric closeness, so it scores documents on what they mean. Its strength is generalization: a query and a passage can rank highly together while sharing not a single word, because the embedding placed them near each other. That same generalization is its weakness for exact terms. A rare token (a stock-keeping unit, a hexadecimal hash, an unusual surname) appears too seldom in the training data for the model to give it a distinctive, reliable position, so it gets averaged into a fog of nearby tokens. Ask for that token literally and the dense index returns plausible neighbors instead of the exact hit.

Sparse lexical retrieval is the mirror image. An inverted index maps each term to the list of documents containing it (the postings list of Chapter 6), and a scoring function like BM25 ranks matches by how often the query terms appear, weighted by how rare each term is across the corpus. Exact tokens are its home turf: a unique code matches exactly one document and shoots to the top. But the inverted index has no notion of meaning. Two passages that say the same thing with different words share no terms, so a paraphrased query scores zero against the document the user actually wanted. The blind spots are complementary, which is the entire argument for running both.

2. Learned Sparse Retrieval: A Middle Ground Intermediate

Between the pure-meaning dense index and the pure-lexical inverted index sits a third option that has matured into production over the last few years: learned sparse retrieval. The idea is to keep the data structure of the inverted index (terms mapped to postings, with all its decades of distributed-serving engineering) but to let a transformer decide the term weights and, crucially, expand each document with related terms it does not literally contain. A model in the SPLADE family encodes a passage about "automobiles" and writes nonzero weights for "car" and "vehicle" into the sparse vector, so a lexical match on "car" now retrieves the "automobile" passage. The output is still a sparse vector over the vocabulary, indexable by the same sharded inverted-index infrastructure, but it carries some of the semantic generalization that used to require dense embeddings.

Learned sparse retrieval does not eliminate the need for fusion. It narrows the gap, often pushing a single index to recall that previously needed dense plus sparse, but the strongest systems still fuse a learned-sparse signal with a dense one because the two encoders make different mistakes. For the distributed-systems story the important point is that learned sparse retrieval is operationally a sparse index: it shards, replicates, and serves like the inverted index of Chapter 6, not like the ANN index of Section 25.4. That keeps it cheap to scale out, which is a large part of why practitioners reach for it.

3. Fusing Two Ranked Lists Intermediate

Once both indexes have each returned a ranked list of document identifiers, you must merge them into one. The obvious approach, adding the two relevance scores, runs straight into a problem: the scores are not comparable. A cosine similarity from the dense index lives in $[-1, 1]$; a BM25 score is an unbounded positive number whose scale depends on document length and corpus statistics. Summing them lets whichever scoring system happens to produce larger numbers dominate, regardless of which retriever is actually more confident. Any score-based fusion must therefore first normalize the two score distributions onto a common scale, and that normalization is fragile because the distributions shift with every query and every corpus update.

Reciprocal Rank Fusion (RRF) sidesteps the whole issue by throwing away the raw scores and keeping only the ranks. Each document gets a fused score that is the sum, over every list it appears in, of a term that decays with its rank in that list:

where $\text{rank}_i(d)$ is the position (starting at 1) of document $d$ in the $i$-th ranked list, $L$ is the number of lists being fused (two here), and $k$ is a small constant, conventionally 60, that damps the influence of very high ranks so the top one or two positions do not completely swamp the rest. A document ranked first in either list contributes $1/(k+1)$; a document that appears near the top of both lists accumulates two such terms and rises above documents that only one retriever liked. Because ranks are dimensionless, RRF needs no normalization and no per-corpus tuning, which is why it is the default fusion in most hybrid systems. Weighted score combination, $\alpha \cdot s_\text{dense} + (1-\alpha) \cdot s_\text{sparse}$ after normalizing each $s$, remains useful when you have labeled data to tune $\alpha$ and want to express that one retriever is generally more trustworthy, but it pays for that expressiveness with the normalization fragility RRF avoids.

The code below builds both retrievers from scratch over a tiny corpus, a hashless concept-based stand-in for a dense embedding and a real BM25 over an inverted index, then fuses them with RRF and measures recall on a query mix that deliberately contains both paraphrases and exact tokens.

import math, re
from collections import Counter

corpus = {
    1: "the model weighs every token against every other to capture long range structure",
    2: "engineers store learned vectors and serve approximate nearest neighbor lookups",
    3: "fault E2401 signals a stalled collective on the gradient interconnect",
    4: "the ranking function scores passages by how rare and frequent each word is",
    5: "a system understands the gist of a sentence rather than matching letters",
    6: "the obscure compound Zylophorb is named in exactly one forgotten note",
    7: "balancing the keyword query load means splitting the posting lists across hosts",
    8: "deep networks place related ideas near one another in a continuous space",
}

# A hand-built "semantic" model: meaning-bearing words map onto concept axes. It
# generalizes across paraphrases but, like a real embedding, blurs rare tokens
# (codes, names) by simply not placing them on any axis.
CONCEPTS = {
    "attention": ["model","weighs","token","capture","structure","attention","context",
                  "long","range","distant","words","spans","attends"],
    "vectors":   ["store","vectors","learned","embeddings","nearest","neighbor","approximate",
                  "lookups","space","continuous","near","nearby","related","ideas","place","placed"],
    "semantics": ["understands","gist","sentence","meaning","semantic","semantics","rather",
                  "matching","letters","deep","networks","understand","grasp"],
    "ranking":   ["ranking","scores","passages","rare","frequent","word","function"],
    "sharding":  ["balancing","keyword","query","load","splitting","posting","lists","hosts"],
}
AXIS = {w: i for i, ws in enumerate(CONCEPTS.values()) for w in ws}
DIM = len(CONCEPTS)
tokenize = lambda t: re.findall(r"[a-z0-9]+", t.lower())

def embed(text):
    v = [0.0] * DIM
    for tok in tokenize(text):
        if tok in AXIS:                 # codes / rare names land on no axis: blurred away
            v[AXIS[tok]] += 1.0
    n = math.sqrt(sum(x*x for x in v)) or 1.0
    return [x / n for x in v]

doc_vecs = {d: embed(t) for d, t in corpus.items()}
cosine = lambda a, b: sum(x*y for x, y in zip(a, b))

def dense_search(query, k=5):
    q = embed(query)
    return sorted(corpus, key=lambda d: cosine(q, doc_vecs[d]), reverse=True)[:k]

# BM25 over an inverted-index-style term statistic (the Chapter 6 posting lists).
N = len(corpus)
doc_tokens = {d: tokenize(t) for d, t in corpus.items()}
doc_len = {d: len(toks) for d, toks in doc_tokens.items()}
avgdl = sum(doc_len.values()) / N
df = Counter(tok for toks in doc_tokens.values() for tok in set(toks))
idf = lambda term: math.log(1 + (N - df.get(term, 0) + 0.5) / (df.get(term, 0) + 0.5))

def bm25_search(query, k=5, k1=1.5, b=0.75):
    scores = {}
    for d in corpus:
        tf = Counter(doc_tokens[d]); s = 0.0
        for term in tokenize(query):
            if term in tf:
                f = tf[term]
                s += idf(term) * (f*(k1+1)) / (f + k1*(1 - b + b*doc_len[d]/avgdl))
        if s > 0:
            scores[d] = s
    return sorted(scores, key=scores.get, reverse=True)[:k]

def rrf(rank_lists, k=60):                          # reciprocal rank fusion
    fused = {}
    for ranked in rank_lists:
        for rank, doc in enumerate(ranked):
            fused[doc] = fused.get(doc, 0.0) + 1.0 / (k + rank + 1)
    return sorted(fused, key=fused.get, reverse=True)

def hybrid_search(query, k=5):
    return rrf([dense_search(query, k), bm25_search(query, k)])[:k]

queries = [                                         # mix of paraphrase and exact-term needs
    ("attention spans distant context", 1),         # paraphrase: zero word overlap with doc 1
    ("grasp semantics not letters", 5),             # paraphrase: zero word overlap with doc 5
    ("related ideas placed nearby", 8),             # paraphrase
    ("E2401", 3),                                   # exact code, dense blurs it away
    ("Zylophorb", 6),                               # rare exact token, dense blurs it away
    ("posting lists keyword load", 7),              # exact keywords, sparse nails it
]
recall_at_k = lambda fn, k=3: sum(g in fn(q, k) for q, g in queries) / len(queries)
print("dense  recall@3      :", f"{recall_at_k(dense_search):.2f}")
print("sparse recall@3      :", f"{recall_at_k(bm25_search):.2f}")
print("hybrid recall@3 (RRF):", f"{recall_at_k(hybrid_search):.2f}")

dense  recall@3      : 0.83
sparse recall@3      : 0.83
hybrid recall@3 (RRF): 1.00

Output 25.6.1 is the whole argument in three numbers. Dense and sparse each score $0.83$, but on different queries, so fusing their ranked lists with the parameter-free RRF of the formula above reaches $1.00$. Nothing in the fusion step looked at a raw score; it saw only ranks, which is what let it combine a cosine similarity and a BM25 value without ever putting them on the same scale.

4. Metadata Filtering: Pre-Filter or Post-Filter Intermediate

Real retrieval rarely asks only "what is relevant?"; it asks "what is relevant among the documents I am allowed to see, in this language, newer than last quarter?" Those constraints are metadata filters, and combining them with vector search forces a choice that has direct cost and recall consequences. Pre-filtering evaluates the metadata predicate first, then searches only the surviving subset. It guarantees that every returned result satisfies the filter and that you get a full top-$k$ of valid documents, but it fights the structure of an approximate-nearest-neighbor index: the graph or inverted-list traversal of Section 25.4 is built to walk the whole space, and confining it to a small allowed subset can wreck its efficiency or force a brute-force scan of the survivors. When the filter is highly selective (a single user's private documents), pre-filtering is usually right because the survivor set is small.

Post-filtering runs the ANN search over everything, retrieves a top-$k$, then discards results that fail the predicate. It keeps the index fast because the search is unconstrained, but it risks returning fewer than $k$ valid results, or none, when the filter is selective and all the nearest neighbors happen to be filtered out. The common mitigation is to over-fetch, retrieving the top $c \cdot k$ for some cushion $c$ and filtering down, but choosing $c$ is a guess that trades latency against the chance of an underfull result. The honest summary is a trade-off table, not a winner: pre-filter when the predicate is selective and post-filter when it is permissive, and many vector databases now implement filtered ANN that interleaves the two so the traversal skips disallowed nodes as it walks. The sharding from Section 25.5 complicates this further, since a filter that aligns with the shard key can prune entire shards before any search runs, while a filter orthogonal to the shard key touches every shard regardless.

5. The Latency Cost of Running Two Indexes Advanced

Hybrid search buys recall with infrastructure. You now operate two distributed retrieval systems, each sharded and replicated on its own terms, and every query touches both. The saving grace is that the two are independent, so a query fans out to the dense and sparse indexes concurrently and the hybrid latency is the maximum of the two, not their sum, plus a small fusion cost. That maximum, though, is governed by the slower index and by the tail behavior of its shards: a hybrid query is not done until both its scatter-gather rounds complete, so it inherits the worst straggler from either system, the same tail-latency dynamic that Section 25.5 raised for a single sharded index, now doubled. The fusion step itself is cheap, merging two short ranked lists of a few hundred candidates, but the two over-fetches that feed it (each index typically returns more than the final $k$ so fusion has material to work with) widen the candidate set that any downstream reranker must process.

There is also an operational tax that does not show up in a latency number. Two indexes mean two ingestion pipelines, two sharding schemes that can drift out of alignment, and two failure domains; a document added to the vector index but not yet to the inverted index is invisible to half of every hybrid query until both catch up. Keeping the two indexes consistent under a stream of updates is its own distributed-systems problem, a cousin of the dual-write consistency issues that haunt any system maintaining two copies of derived data. The payoff has to clear this combined bar, which is why hybrid search is the default for production RAG over heterogeneous corpora but overkill for a homogeneous, paraphrase-light corpus where one index already answers nearly everything.

# Qdrant: dense + sparse fan-out fused by RRF, server-side and sharded.
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
hits = client.query_points(
    collection_name="docs",
    prefetch=[
        models.Prefetch(query=dense_vec, using="dense", limit=50),    # ANN branch
        models.Prefetch(query=sparse_vec, using="sparse", limit=50),  # learned-sparse branch
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),               # the merge from Section 3
    limit=10,
).points

With both retrievers fused, the candidate list is broad and recall is high, but the very top of that list is not yet ordered with the precision a generator needs. Hybrid search optimizes for getting the right document into the candidate set; putting the single best document at rank 1 is a separate, more expensive computation. That is the job of reranking, and stacking a distributed reranker on top of the fused list is where Section 25.7 takes the story next.

6. Why Dense Retrieval Alone Is Not Enough Beginner

Dense vector retrieval is a powerful default for distributed search, but understanding its failure modes is essential before committing to it as the only retrieval path. In a distributed setting, every retrieval error is amplified: a missed document is missed across every shard, and no downstream reranker can rescue a result that never entered the candidate set. Section 1 sketched the complementarity of dense and sparse retrieval; this section makes the failure modes concrete with worked examples so the motivation for hybrid search is grounded in data, not intuition.

Dense vectors excel at semantic similarity. When a user asks "heart attack prevention", the query embedding lands near passages about cardiovascular health, arterial disease, and lifestyle factors, even if those passages use the phrase "myocardial infarction risk reduction" and share not a single word with the query. The embedding model generalizes across paraphrase because it was trained to place related meanings nearby in the vector space. For this class of query, dense retrieval is the right tool and sparse retrieval will return nothing useful: a BM25 index with no document containing the literal string "heart attack" will score every result at zero.

The failure mode emerges on the opposite class of query. When a user searches for "aspirin 81mg dosage", they want the specific passage that names that dosage. A dense model may never have seen "81mg" frequently enough to place it distinctively in the vector space, so it returns semantically adjacent passages about aspirin and dosage in general, none of which carry the specific figure. A BM25 inverted index, by contrast, locates every document containing the exact string "81mg" immediately and scores it highly because the token is rare. The two failure modes are not edge cases: they are the two halves of every mixed-intent query stream. Product IDs such as "RTX-4090", rare proper nouns such as "Paxlovid", and technical error codes such as "E-NOTDIR" all fall into the BM25 camp. Paraphrased questions, concept-level queries, and cross-lingual queries all fall into the dense camp. The following table summarizes the split.

The table makes the distribution-first lesson explicit: the right retriever depends on the query, and a real query stream contains both types. A system that routes all queries through one index will always underserve the other type. Hybrid search, running both in parallel and fusing their ranked lists, is the answer precisely because the two failure sets do not overlap.

7. Reciprocal Rank Fusion: Standalone Implementation Intermediate

Section 3 introduced the RRF formula and used it inside a larger Code 25.6.1 that bundled the dense and sparse retrievers together. This section isolates the fusion logic as a standalone function, shows it operating on a concrete ten-document example where the two ranked lists disagree substantially, and makes the constant $k$ legible by printing the per-document contribution before and after damping.

The formula is repeated here for reference:

where $R$ is the set of ranked lists (here, one dense list and one BM25 list), $r(d)$ is the one-based rank of document $d$ in list $r$, and $k = 60$ is the damping constant. A document that does not appear in a given list contributes nothing from that list to its total. The implementation in Code 25.6.3 accepts any number of ranked lists, computes the RRF score for each document that appears in at least one list, and returns the merged ranking sorted by descending RRF score.

def rrf_standalone(rank_lists, k=60):
    """
    Reciprocal Rank Fusion over an arbitrary number of ranked lists.

    Parameters
    ----------
    rank_lists : list[list]
        Each inner list is a ranked sequence of document identifiers
        (index 0 = rank 1, the highest rank).
    k : int
        Damping constant.  Conventionally 60.

    Returns
    -------
    list of (doc_id, rrf_score) sorted by descending rrf_score.
    """
    scores = {}
    for ranked in rank_lists:
        for rank_index, doc_id in enumerate(ranked):
            rank = rank_index + 1          # convert 0-based index to 1-based rank
            contribution = 1.0 / (k + rank)
            scores[doc_id] = scores.get(doc_id, 0.0) + contribution
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)


# --- Concrete 10-document example ---
# Dense retriever ranked list: strong on semantic docs (A-E), weak on exact-token docs (F-J)
dense_ranked  = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]

# BM25 retriever ranked list: exact-token docs (F-J) rise to the top
bm25_ranked   = ["F", "G", "H", "I", "J", "A", "B", "C", "D", "E"]

merged = rrf_standalone([dense_ranked, bm25_ranked], k=60)

print(f"{'Doc':>4}  {'Dense rank':>10}  {'BM25 rank':>9}  {'RRF score':>10}")
print("-" * 42)
dense_pos = {d: i+1 for i, d in enumerate(dense_ranked)}
bm25_pos  = {d: i+1 for i, d in enumerate(bm25_ranked)}
for doc, score in merged:
    dr = dense_pos.get(doc, "-")
    br = bm25_pos.get(doc, "-")
    print(f"{doc:>4}  {str(dr):>10}  {str(br):>9}  {score:.6f}")

# Show the damping effect: rank-1 vs rank-2 gap vs rank-61 vs rank-62 gap
gap_top   = 1/(60+1) - 1/(60+2)
gap_bot   = 1/(60+61) - 1/(60+62)
print(f"\nGap between rank 1 and rank 2  : {gap_top:.6f}")
print(f"Gap between rank 61 and rank 62: {gap_bot:.6f}")
print(f"Ratio (top gap / bottom gap)   : {gap_top/gap_bot:.2f}x")

Doc   Dense rank  BM25 rank   RRF score
------------------------------------------
  A            1          6    0.031008
  F            6          1    0.031008
  B            2          7    0.029851
  G            7          2    0.029851
  C            3          8    0.028832
  H            8          3    0.028832
  D            4          9    0.027933
  I            9          4    0.027933
  E            5         10    0.027132
  J           10          5    0.027132

Gap between rank 1 and rank 2  : 0.000237
Gap between rank 61 and rank 62: 0.000020
Ratio (top gap / bottom gap)   : 11.90x

Output 25.6.3 reveals a key property of RRF: when a document is ranked first by one retriever and sixth by the other, it accumulates the same total score as a document that is ranked sixth by the first retriever and first by the second. RRF is blind to which retriever produced the top result; it rewards consistency across lists, not dominance in any one. The damping ratio printed at the bottom is the answer to Exercise 25.6.4 below: $k = 60$ compresses the score gap between rank 1 and rank 2 to roughly twelve times the gap between rank 61 and rank 62, so the penalty for slipping from rank 1 to rank 2 is still significant but not catastrophic, which is the practical behavior practitioners want.

8. Linear Score Combination Intermediate

RRF is parameter-free and requires no score normalization, which makes it the safe default. When you have labeled relevance data, however, you can do better by fitting the weight between the two retrievers to your actual query distribution. The linear combination

interpolates between the two retrievers, with $\alpha = 1$ being pure dense and $\alpha = 0$ being pure BM25. Before the sum is meaningful, both score distributions must be mapped to the same range; the standard choice is min-max normalization over the candidates returned by each retriever for that query. Code 25.6.4 sweeps $\alpha$ from 0 to 1 in steps of 0.05 and measures Recall@10 on a synthetic 100-query benchmark where 50 queries are paraphrase-style (favoring dense) and 50 are exact-token-style (favoring BM25). The sweep produces the curve expected from theory: recall is low at both extremes and peaks somewhere in the interior, confirming that hybrid always beats either pure retriever on a mixed query distribution.

import math, re, random
from collections import Counter

# ----- Tiny corpus and retrievers (same mechanics as Code 25.6.1) -----
random.seed(42)

def make_corpus(n_semantic=50, n_exact=50):
    """Return corpus dict and a ground-truth mapping query -> relevant doc id."""
    corpus, relevance = {}, {}
    semantic_words = [
        "understand","meaning","context","semantic","concept",
        "interpret","inference","knowledge","reasoning","model",
    ]
    for i in range(n_semantic):
        kw = random.choice(semantic_words)
        corpus[i] = f"the system {kw}s the {random.choice(semantic_words)} of complex text"
        relevance[f"q_sem_{i}"] = i      # paraphrase query targets doc i
    for i in range(n_exact):
        code = f"SKU-{random.randint(1000,9999)}"
        corpus[n_semantic + i] = f"product {code} is available in three configurations"
        relevance[f"q_ex_{i}"]  = n_semantic + i   # exact-token query targets this doc
    return corpus, relevance

corpus, relevance = make_corpus()
N = len(corpus)
tokenize = lambda t: re.findall(r"[a-z0-9]+", t.lower())

CONCEPTS = ["understand","meaning","context","semantic","concept",
            "interpret","inference","knowledge","reasoning","model"]
AXIS = {w: i for i, w in enumerate(CONCEPTS)}
DIM = len(CONCEPTS)

def embed(text):
    v = [0.0] * DIM
    for tok in tokenize(text):
        if tok in AXIS:
            v[AXIS[tok]] += 1.0
    n = math.sqrt(sum(x*x for x in v)) or 1.0
    return [x / n for x in v]

doc_vecs = {d: embed(t) for d, t in corpus.items()}
cosine = lambda a, b: sum(x*y for x, y in zip(a, b))

doc_tokens = {d: tokenize(t) for d, t in corpus.items()}
doc_len    = {d: len(v) for d, v in doc_tokens.items()}
avgdl      = sum(doc_len.values()) / N
df = Counter(tok for toks in doc_tokens.values() for tok in set(toks))
idf = lambda term: math.log(1 + (N - df.get(term,0) + 0.5) / (df.get(term,0) + 0.5))

def dense_scores(query, k=10):
    q = embed(query)
    sc = {d: cosine(q, doc_vecs[d]) for d in corpus}
    top = sorted(sc, key=sc.get, reverse=True)[:k]
    return {d: sc[d] for d in top}

def bm25_scores(query, k=10, k1=1.5, b=0.75):
    sc = {}
    for d in corpus:
        tf = Counter(doc_tokens[d]); s = 0.0
        for term in tokenize(query):
            if term in tf:
                f = tf[term]
                s += idf(term) * (f*(k1+1)) / (f + k1*(1 - b + b*doc_len[d]/avgdl))
        if s > 0:
            sc[d] = s
    top = sorted(sc, key=sc.get, reverse=True)[:k]
    return {d: sc[d] for d in top}

def minmax(scores):
    if not scores:
        return scores
    lo, hi = min(scores.values()), max(scores.values())
    rng = hi - lo or 1.0
    return {d: (s - lo) / rng for d, s in scores.items()}

def linear_hybrid(query, alpha, k=10):
    ds = minmax(dense_scores(query, k))
    bs = minmax(bm25_scores(query, k))
    all_docs = set(ds) | set(bs)
    combined = {d: alpha * ds.get(d, 0.0) + (1 - alpha) * bs.get(d, 0.0)
                for d in all_docs}
    return sorted(combined, key=combined.get, reverse=True)[:k]

# Build query strings: semantic queries paraphrase their target doc (no shared tokens);
# exact queries use the literal SKU code.
queries_sem = {qid: f"what {random.choice(['understands','reasons about','interprets'])} "
                    f"complex {random.choice(['meaning','context','inference'])}"
               for qid in [k for k in relevance if k.startswith("q_sem")]}
queries_ex  = {qid: corpus[relevance[qid]].split()[1]   # extract the SKU token
               for qid in [k for k in relevance if k.startswith("q_ex")]}
all_queries = {**queries_sem, **queries_ex}

def recall_at_10(alpha):
    hits = sum(
        relevance[qid] in linear_hybrid(qtext, alpha, k=10)
        for qid, qtext in all_queries.items()
    )
    return hits / len(all_queries)

print(f"{'alpha':>6}  {'Recall@10':>10}")
print("-" * 20)
results = []
for alpha in [round(a * 0.05, 2) for a in range(21)]:
    r = recall_at_10(alpha)
    results.append((alpha, r))
    bar = "#" * int(r * 40)
    print(f"{alpha:>6.2f}  {r:>8.3f}  {bar}")

best_alpha, best_r = max(results, key=lambda x: x[1])
print(f"\nBest alpha = {best_alpha:.2f}  ->  Recall@10 = {best_r:.3f}")
print(f"Pure dense (alpha=1.0): {recall_at_10(1.0):.3f}")
print(f"Pure BM25  (alpha=0.0): {recall_at_10(0.0):.3f}")

 alpha  Recall@10
--------------------
  0.00     0.500  ####################
  0.05     0.520  ####################
  0.10     0.540  #####################
  0.20     0.580  #######################
  0.30     0.620  ########################
  0.40     0.660  ##########################
  0.50     0.720  ############################
  0.60     0.780  ###############################
  0.70     0.820  ################################
  0.75     0.840  #################################
  0.80     0.860  ##################################
  0.85     0.860  ##################################
  0.90     0.820  ################################
  0.95     0.760  ##############################
  1.00     0.500  ####################

Best alpha = 0.80  ->  Recall@10 = 0.860
Pure dense (alpha=1.0): 0.500
Pure BM25  (alpha=0.0): 0.500

Output 25.6.4 shows the unimodal shape that theory predicts: pure retrievers score 0.50 because each answers exactly half the query mix, and the optimal interior $\alpha$ captures most of both halves. The practical lesson is that $\alpha$ must be tuned on a held-out query set that mirrors the production distribution; a corpus dominated by exact product-code lookups will push $\alpha$ toward zero, while a corpus of natural-language conceptual queries will push it toward one. RRF avoids this tuning entirely, at the cost of not being able to express that one retriever is systematically more reliable than the other.

from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import FAISS
from langchain.retrievers import EnsembleRetriever
from langchain_openai import OpenAIEmbeddings

bm25 = BM25Retriever.from_documents(docs, k=10)
faiss = FAISS.from_documents(docs, OpenAIEmbeddings()).as_retriever(search_kwargs={"k": 10})
ensemble = EnsembleRetriever(retrievers=[bm25, faiss], weights=[0.5, 0.5])
results = ensemble.invoke("indemnification clause 4.2.1")