Problem Definition

"One user pressed Enter. By the time the answer came back, I had fanned the question out across a thousand shards, gathered their best guesses, and reassembled myself before anyone noticed I was never in one place to begin with."

A Single Query, About to Fan Out Across a Thousand Shards

Retrieval-augmented generation, or RAG, answers a question by first finding relevant passages in a corpus and then conditioning a language model on those passages while it writes the answer. On a laptop, over a few thousand documents, RAG is a weekend project: embed the documents, drop the vectors into an in-memory index, and call a model. The interesting version, and the one this chapter studies, is the version that serves the open web. When the corpus is measured in billions of pages, refreshed continuously, and queried by a live user population, every component that was trivial on the laptop becomes a distributed system in its own right. The purpose of this case study is to show how the machinery built in Parts II through VII assembles into one coherent product, and where the seams between those parts actually fall.

We will treat the system as a concrete target rather than an abstraction. Throughout the chapter, the running specification is a web-scale RAG service with a corpus on the order of ten billion documents, a freshness requirement measured in hours for the most active slice of the web, a query path that must return grounded answers within a latency a user tolerates, and a cost envelope that rules out brute force. Those four pressures, corpus size, freshness, latency and throughput, and cost, are the requirements; the rest of the chapter is the design that satisfies them.

1. The Requirements: Four Numbers That Rule Out One Machine Beginner

A problem definition is only useful if it commits to numbers, because the numbers decide which ceilings bind and therefore which axes of distribution we must reach for. We fix four requirements and carry them through the entire chapter. The first is corpus size: the system indexes on the order of $N_{\text{docs}} = 10^{10}$ documents, each split into a handful of retrievable passages. The second is freshness: a meaningful fraction of the web changes daily, and the most active pages must be re-crawled and re-indexed within hours, not weeks, so the pipeline is never a one-time batch job but a continuously running flow. The third is the service-level objective, or SLO, on the query path: the system must sustain a throughput measured in thousands of queries per second while keeping per-query latency inside a budget on the order of a second, including both retrieval and generation. The fourth is cost: the design must hit the first three at a price that a real operator would pay, which immediately disqualifies any approach that scans the whole corpus per query or re-embeds the whole corpus per update.

These four requirements interact, and the interaction is the whole subject. Freshness fights cost, because re-embedding more often burns more accelerator time. Latency fights corpus size, because a larger index means more candidates to search per query. Throughput fights latency, because every concurrent query competes for the same retrieval and generation hardware. A single machine fails all four at once and for independent reasons: it cannot store the corpus, cannot embed it in any reasonable time, cannot hold the index, and cannot serve the request volume. The next section makes that failure quantitative.

2. Why a Single Machine Is Not Even Close Beginner

The argument for distribution here is not rhetorical; it falls out of arithmetic on the four requirements. Take the corpus of $N_{\text{docs}} = 10^{10}$ documents, split each into a few passages, and embed every passage into a vector of dimension $d$ stored in half precision. The number of vectors to store and search is

where $c$ is passages per document, $d$ is the embedding dimension, and $b$ is bytes per dimension. With $c = 3$, $d = 1024$, and $b = 2$ bytes, the index alone runs to tens of terabytes of raw vectors, far past the memory of any single node. The number of shards the index must be split into is the index size divided by the memory each node can devote to it,

which for a node offering tens of gigabytes of fast memory to the index lands in the hundreds. The compute to build the index once is equally decisive. If embedding costs a fixed throughput $r$ passages per second on one accelerator, the total one-pass embedding work is

independent of how we parallelize it; parallelizing only divides the wall-clock by the fleet size. Finally, the query path obeys Little's law: the number of queries in flight at any instant is the arrival rate times how long each one stays in the system,

so a throughput $\lambda$ of thousands of queries per second at a latency $L$ near a second means thousands of queries are being served concurrently, which no single server's compute or memory can hold. Figure 36.1.1 shows the four pressures and the stages they fall on; Code 36.1.1 puts real numbers behind every one of these formulas.

The diagram fixes the vocabulary for the chapter. The top row is the offline build path that runs continuously to satisfy the freshness requirement; the middle row is the online query path that runs per request under the latency SLO; the dashed bottom loop is the evaluation that keeps both honest. The numbers on each box come from the same back-of-envelope model, which we now run with real arithmetic so that no estimate in this chapter is a guess.

import math

N_docs            = 10_000_000_000   # corpus size (requirement 1)
chunks_per_doc    = 3                 # retrievable passages per document
bytes_per_chunk   = 1024             # cleaned text per passage, bytes
dim               = 1024             # embedding dimension d
bytes_per_dim     = 2                # fp16 vector storage, b
embed_rate        = 2000             # passages/sec on one accelerator, r
mem_per_node_gb   = 80              # memory one node devotes to the index

n_vec   = N_docs * chunks_per_doc                       # total vectors N_vec
text_tb = n_vec * bytes_per_chunk / 1e12               # cleaned text storage
S_index = n_vec * dim * bytes_per_dim                  # index size in bytes
vec_tb  = S_index / 1e12
shards  = math.ceil(S_index / (mem_per_node_gb * 1e9)) # K_shard
gpu_h   = n_vec / embed_rate / 3600                    # T_embed in GPU-hours

print(f"total vectors N_vec        : {n_vec:,}")
print(f"cleaned text storage       : {text_tb:,.1f} TB")
print(f"fp16 vector index S_index  : {vec_tb:,.1f} TB")
print(f"index shards K_shard       : {shards:,}")
print(f"one-pass embed T_embed     : {gpu_h:,.0f} GPU-hours")

for fleet in (1, 1000):                                 # wall-clock = T_embed / fleet
    print(f"  embed wall-clock on {fleet:>4} GPUs : {gpu_h/fleet:>10,.1f} h "
          f"({gpu_h/fleet/24:,.1f} days)")

for qps, lat in ((5000, 0.8), (20000, 0.8)):            # Little's law: C = lambda * L
    print(f"  QPS={qps:>6}, latency={lat}s -> queries in flight C = {qps*lat:,.0f}")

total vectors N_vec        : 30,000,000,000
cleaned text storage       : 30.7 TB
fp16 vector index S_index  : 61.4 TB
index shards K_shard       : 768
one-pass embed T_embed     : 4,167 GPU-hours
  embed wall-clock on    1 GPUs :    4,166.7 h (173.6 days)
  embed wall-clock on 1000 GPUs :        4.2 h (0.2 days)
  QPS=  5000, latency=0.8s -> queries in flight C = 4,000
  QPS= 20000, latency=0.8s -> queries in flight C = 16,000

The output settles the question. The single accelerator needs about $173$ days to embed the corpus once, which alone violates the freshness requirement before any user has typed a query. The index is over sixty terabytes and demands on the order of $768$ shards. The query path keeps thousands of requests in flight simultaneously. None of these is a marginal overshoot that a faster chip would absorb; each is one to three orders of magnitude beyond a single node, and each lands on a different stage of the pipeline. That separation is exactly what makes the six-axis decomposition the right tool.

3. Decomposing the Pipeline Onto the Six Axes Intermediate

The six axes of distribution from Section 1.1, distribute data, distribute training, distribute the model, distribute inference, coordinate the cluster, and distribute intelligence, give us a map onto which every stage of Figure 36.1.1 places cleanly. Mapping the stages is the planning act of the whole chapter: it tells us which earlier part of the book owns each stage and which later section of this chapter develops it. Table 36.1.1 is that map, and it doubles as the chapter's table of contents.

Reading the table top to bottom traces the dataflow of Figure 36.1.1, and reading the third column traces the path back through the book. The crawl and the cleaning stages are pure data distribution: they are MapReduce and Spark jobs over a petabyte-class stream, owned by Chapter 6 and Chapter 7, with the storage and loading machinery of Chapter 8 underneath. Embedding is distributed inference applied to a fixed corpus rather than a live request stream, which is why it borrows the encoder and batching techniques of Chapter 15. Sharding the index is distributing a structure too large for one node, the same partitioning logic that splits a model across devices, here applied to the vector store of Chapter 25. Retrieval and ranking are a scatter-gather over those shards, a cluster-coordination problem. Generation is distributed LLM serving from Chapter 24, multiplied across the fleet to meet the throughput SLO. Orchestration that holds the per-query flow together is the agent-orchestration material of Chapter 32. Evaluation closes the loop. No stage is new machinery; the contribution of this chapter is the assembly and the seams.

4. The Build Path Versus the Query Path Intermediate

One structural decision organizes the rest of the chapter, and it is visible as the two horizontal bands in Figure 36.1.1. The offline build path and the online query path have opposite cost profiles and must be engineered against opposite metrics. The build path is throughput-bound and latency-tolerant: it processes the whole corpus, so it is measured in total accelerator-hours and dollars, and a passage embedded a few minutes late costs nothing. The query path is latency-bound and throughput-replicated: each request must finish inside the SLO, so it is measured in tail latency and in how many concurrent requests the fleet sustains, and being a few seconds slow costs a user. Conflating these two paths is the most common design error in real RAG systems, because the techniques that make the build path cheap, enormous batches and relaxed deadlines, are exactly the techniques that ruin the query path.

The two paths meet at one artifact: the sharded index. The build path produces it; the query path consumes it. Freshness is precisely the rate at which updates flow from the build path into the index that the query path is already serving, which is why the index cannot be a static file but must be a live, incrementally updated structure, a point Section 36.5 takes up in detail. The concurrency the query path must hold, the $C = \lambda L$ from Output 36.1.1, is the budget that Section 36.7 spends on replicated generation. Keeping the two paths logically separate while letting them share exactly one artifact is the architectural spine of the chapter.

# pip install langchain-community faiss-cpu sentence-transformers
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import HuggingFaceEmbeddings

emb   = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
store = FAISS.from_texts(documents, emb)          # embeds + indexes, all in RAM
hits  = store.similarity_search(query, k=5)        # exact scan over the whole index
answer = llm(prompt_with(hits))                    # one model call, one machine

5. What This Chapter Builds, Section by Section Beginner

With the requirements fixed, the scale proven, and the axes mapped, the path through the chapter is set. Section 36.2 builds the distributed crawler that produces the raw corpus. Section 36.3 cleans and deduplicates that corpus at scale, the stage that most determines final answer quality. Section 36.4 builds the sharded lexical index over the cleaned corpus, the term-to-document structure too large for one node. Section 36.5 embeds the cleaned passages across a fleet, spending the $T_{\text{embed}}$ budget of Output 36.1.1 and writing the resulting vectors into the live index, the artifact both paths share. Section 36.6 builds the fan-out retrieval and ranking that turn a query into candidate passages. Section 36.7 conditions a replicated, sharded generator on those passages to produce grounded answers inside the latency SLO, and orchestrates the per-query flow. Section 36.8 closes the loop with end-to-end evaluation of retrieval quality, faithfulness, and cost. Section 36.9 hands the pipeline back as a staged project that grows a single-machine baseline into the full distributed system one milestone at a time. Each section opens with the slice of Figure 36.1.1 it owns and the requirement from Section 1 it must satisfy.