Evaluating Distributed Retrieval

"They told the dashboard I answered in two milliseconds. They forgot to mention I found the wrong document."

An ANN Index Tuned a Little Too Aggressively

Across this chapter we built a retrieval pipeline out of distributed parts: embeddings computed on a fleet of encoders, vectors stored in a sharded database, an approximate index that answers nearest-neighbor queries in sublinear time, replicas that absorb load, a scatter-gather that fans a query across shards, a reranker that sharpens the top results, and caches that short-circuit the repeats. Each part has a dial, and turning a dial moves both quality and cost. The final engineering question is therefore not "is it correct?" but "where on the trade-off surface does it sit, and is that the point we want?" Answering it requires measuring the system along two axes that most teams are tempted to measure separately, and that is exactly the temptation this section exists to break.

1. Two Scoreboards: Quality and Systems Beginner

The first axis is retrieval quality: of the items the system returned, how many were the ones it should have returned, and were the best ones near the top? The second axis is systems behavior: how many queries per second can the fleet sustain, what is the tail latency of a query, how long does the index take to build, how much memory does it occupy, and what does a single query cost. A retrieval system is healthy only when both scoreboards are healthy at once, and the central lesson of distributed retrieval is that the two are linked by the very approximations that make scale-out possible. You cannot tune one in isolation without silently moving the other.

This is the construct-matching point from Chapter 5 in its sharpest form. A metric is only meaningful next to the configuration that produced it. Recall measured at one approximate-search setting and latency measured at another describe two different systems and cannot be compared; they must be co-measured in one pass on one configuration. The rest of this section names the metrics on each scoreboard, then shows how to bind them together.

2. Retrieval-Quality Metrics Beginner

The headline quality metric for distributed retrieval is recall@k, because approximate search exists precisely to trade recall for speed. Let $R_q$ be the set of truly relevant (or, for ANN, the exact-nearest) items for query $q$, and let $\hat{R}_q^{(k)}$ be the top $k$ the system returned. Recall@k averages, over a query set $Q$, the fraction of the true items that were recovered,

For an ANN index the natural ground truth is the exact brute-force top-$k$, so recall@k measures how much of the exact answer the approximation kept; a recall of $0.9$ at $k=10$ means the index recovered nine of the ten exact neighbors on average. Precision@k, the complementary fraction of returned items that are relevant, matters more when retrieval feeds a downstream consumer with a fixed budget of slots. When the order within the top $k$ matters, as it does when a reranker or a generator reads the list top-down, ranking-aware metrics take over: nDCG discounts a relevant item by the logarithm of its rank so that a correct answer at position one counts more than the same answer at position ten, and MRR rewards getting the single first relevant item as high as possible. These are the same ranking metrics that govern reranking quality in Section 25.7, now read as evaluation rather than as a training objective.

Recall, precision, and nDCG measure the retriever in isolation. When retrieval feeds a generator, as it does in retrieval-augmented generation, the metric that ultimately matters is the quality of the generated answer, which depends on retrieval only through the context it supplies. Two end-to-end measures dominate here: groundedness (is every claim in the answer supported by a retrieved passage?) and faithfulness (does the answer avoid asserting anything the retrieved context does not support?). Both are commonly scored with an LLM-as-judge that reads the answer, the context, and the question and rates support, a technique we treat with its calibration caveats in Chapter 5. The crucial structural fact is that a retrieval change can lift recall yet leave answer quality flat (the generator already had enough context) or, worse, raise recall of near-duplicates that crowd out the one passage the answer needed. Quality must therefore be measured at the layer the user actually experiences, not only at the retriever's own output.

3. Systems Metrics Intermediate

The systems scoreboard begins with query throughput (QPS), the sustained rate the fleet serves under a fixed latency ceiling, and tail latency, reported at the p99 rather than the mean because a user feels the slow queries, not the average ones. Tail latency is where the distributed structure of the system shows itself most plainly: a query fanned across $S$ shards in the scatter-gather of Section 25.5 is only as fast as its slowest shard, so the p99 of the merged query is governed by the p99 of the worst replica, not the typical one. This is the fan-out tail-amplification effect, and it is why adding shards to raise capacity can quietly raise tail latency even as it lowers mean latency.

Two offline systems metrics round out the picture. Index build time and memory bound how often you can rebuild the index as the corpus grows and how many vectors fit per node; a graph index such as HNSW is fast to query but heavy to build and to hold in memory, while a quantized IVF-PQ index is lighter in memory but recovers less recall, the same construction trade-offs catalogued in Section 25.4. Finally, cost-per-query ties the systems scoreboard to money: it folds the accelerator and memory time a query consumes into a single figure that, tracked over the fleet, is the number a budget owner cares about. Cost-per-query is the retrieval instance of the cost analysis of Section 5.5, and it is the third axis of the frontier this section is building toward.

4. The Coupling: Report Recall and Latency Together Intermediate

The two scoreboards are not independent dashboards to be read side by side; they are a single surface. The cleanest way to see this is to sweep the recall dial and watch latency and cost move with it. The demonstration below builds a small IVF index in pure Python, computes exact brute-force ground truth, and then for each nprobe setting measures recall@10, the p99 latency of the scatter-gather query, and a derived cost-per-query, all co-measured in one pass on one configuration so the numbers are construct-matched by construction. The joint operating point we care about is the cheapest configuration that meets a recall target,

where $\mathcal{C}$ is the set of configurations (here, the nprobe values) and $\tau$ is the recall floor the application demands. Reporting only $\min_c \text{latency}(c)$ would select the fastest configuration regardless of whether it found anything useful, which is exactly the misleading headline this section warns against.

import math, random
random.seed(7)

DIM, N, NQ, NLIST, K = 16, 4000, 200, 64, 10        # dims, corpus, queries, IVF cells, recall@K

def rand_vec(): return [random.gauss(0, 1) for _ in range(DIM)]
def dist2(a, b): return sum((x - y) ** 2 for x, y in zip(a, b))

corpus  = [rand_vec() for _ in range(N)]
queries = [rand_vec() for _ in range(NQ)]
centroids = random.sample(corpus, NLIST)             # coarse quantizer

def nearest_cell(v): return min(range(NLIST), key=lambda i: dist2(v, centroids[i]))
cells = [[] for _ in range(NLIST)]
for idx, v in enumerate(corpus): cells[nearest_cell(v)].append(idx)

# exact ground truth: brute-force top-K neighbours per query
ground_truth = [set(sorted(range(N), key=lambda j: dist2(q, corpus[j]))[:K]) for q in queries]

def ivf_search(q, nprobe):                           # search nprobe nearest cells
    probed = sorted(range(NLIST), key=lambda i: dist2(q, centroids[i]))[:nprobe]
    cand = [j for ci in probed for j in cells[ci]]
    work = NLIST + len(cand)                          # distance evals ~ compute ~ cost
    return set(sorted(cand, key=lambda j: dist2(q, corpus[j]))[:K]), work

BASE_MS, PER_EVAL_US, TAIL, COST_PER_MS = 1.2, 0.9, 1.35, 0.00004
print(f"{'nprobe':>6} {'recall@10':>10} {'p99_ms':>8} {'$/query':>10}")
rows = []
for nprobe in [1, 2, 4, 8, 16, 32, 64]:
    hits, lat = 0, []
    for q, gt in zip(queries, ground_truth):
        found, work = ivf_search(q, nprobe)
        hits += len(found & gt)
        lat.append((BASE_MS + work * PER_EVAL_US / 1000.0) * TAIL)   # p99 straggler inflation
    recall = hits / (NQ * K); lat.sort()
    p99 = lat[int(0.99 * (len(lat) - 1))]; cost = p99 * COST_PER_MS
    rows.append((nprobe, recall, p99, cost))
    print(f"{nprobe:>6} {recall:>10.3f} {p99:>8.2f} {cost:>10.5f}")

TARGET = 0.90
knee = min([r for r in rows if r[1] >= TARGET], key=lambda r: r[2])
fast = min(rows, key=lambda r: r[2])
print(f"cheapest at recall>={TARGET}: nprobe={knee[0]} (recall={knee[1]:.3f}, p99={knee[2]:.2f} ms)")
print(f"fastest overall:          nprobe={fast[0]} (recall={fast[1]:.3f}): misleading alone")

nprobe  recall@10   p99_ms    $/query
     1      0.266     2.04    0.00008
     2      0.408     2.28    0.00009
     4      0.583     2.55    0.00010
     8      0.771     3.10    0.00012
    16      0.930     4.01    0.00016
    32      0.997     5.28    0.00021
    64      1.000     6.56    0.00026
cheapest at recall>=0.9: nprobe=16 (recall=0.930, p99=4.01 ms)
fastest overall:          nprobe=1 (recall=0.266): misleading alone

The numbers tell the whole story of the section. Read the p99 column alone and nprobe 1 looks like a triumph at two milliseconds; read recall@10 beside it and that triumph collapses to a system that misses three of every four exact neighbors. The honest summary of this index is not a number but a curve: the frontier traced by the rows of Output 25.9.1, on which every team must choose a point. Figure 25.9.1 draws that frontier so the shape of the trade-off is visible at a glance.

5. Online Evaluation, A/B Testing, and Embedding Drift Advanced

Offline metrics on a fixed query set tell you how a change behaves on yesterday's data; they cannot tell you how real users respond to it. For that you need online evaluation: route a fraction of live traffic to the new retriever in an A/B test and compare downstream outcomes (click-through, answer acceptance, task success) against the control, with the experiment infrastructure and shadow-deployment machinery that Chapter 26 develops for the whole fleet. A retrieval change that improves offline recall but degrades the live acceptance rate is a real and common outcome, usually because the offline relevance labels and the live objective measure different things, the construct mismatch of Chapter 5 reappearing online.

The harder, slower failure is drift. Embeddings are trained on a snapshot of the world; as the corpus and the queries evolve, the geometry the index was built on stops matching the geometry the queries now inhabit, and recall decays even though no code changed. Detecting this requires monitoring the distribution of query and document embeddings over time and alerting when it shifts, the same drift-detection problem posed for streaming features in Section 9.9, now applied to the embedding space rather than to raw features. The remedy, re-embedding the corpus and rebuilding the index with a refreshed model, is expensive (it is the index-build cost from Section 3 paid in full), which is exactly why drift must be measured continuously: you rebuild when the evidence says recall has decayed, not on a fixed calendar that either wastes compute or lets quality rot.

# pip install beir ragas
from beir.retrieval.evaluation import EvaluateRetrieval
# results[qid][doc_id] = score, from YOUR sharded retriever, measured WITH its p99
ndcg, _map, recall, precision = EvaluateRetrieval.evaluate(qrels, results, k_values=[10])
print(recall["Recall@10"])          # quality axis from the library

from ragas import evaluate
from ragas.metrics import faithfulness, context_precision
rag_scores = evaluate(dataset, metrics=[faithfulness, context_precision])   # answer-quality axis
# the latency / cost-per-query axis you still measure yourself, in the same run

6. Chapter Summary Beginner

This chapter built retrieval-augmented generation as a distributed pipeline and then taught you to evaluate it as one. The arc ran from the embeddings produced on a fleet of encoders (Section 25.2), through the vector databases that store them (Section 25.3) and the approximate-nearest-neighbor index families that search them in sublinear time (Section 25.4), to the sharding and replication that let the index scale and the scatter-gather that fans a query across shards with its tail-latency cost (Section 25.5). On top of that base we layered hybrid search that fuses lexical and dense signals (Section 25.6), retrieve-then-rerank cascades that sharpen the top results (Section 25.7), and the layered caches that absorb the repeats (Section 25.8). This final section closed the loop: a retrieval system is evaluated on recall and latency jointly, never on either alone, because the approximations that buy scale-out also set the recall, and a fast retriever at low recall is a fast wrong answer.