System Architecture

"A request, touching twelve services and returning before you notice. I carry no opinions of my own; I just make sure the slow one never gets to speak last."

A Latency Budget That Has Seen Every Stage Run Late

Every prior section of this chapter solved one problem in isolation: how to shard a billion-row embedding table, how to retrieve a few hundred candidates from a billion items, how to fetch features without a per-request database storm, how to score candidates with a deep model, how to apply diversity and policy rules to the final list. Each was correct on its own bench. The architecture is what happens when you must run all of them, for one request, at peak traffic, while a network link is flapping and a ranking replica is restarting. The unifying constraint is a number the product team gives you and will not negotiate: the recommendation must arrive within, say, 120 milliseconds at the 99th percentile, because beyond that the slot on the page is filled with a fallback and the user has already scrolled. Everything below is organized around spending that 120 milliseconds wisely across machines that each want more of it than they can have.

1. The Request Path: One User, Five Fleets Beginner

Follow a single recommendation request through Figure 38.8.1. It arrives at the gateway, which authenticates the user, attaches the request context (device, locale, session), and routes to a recommender endpoint. The gateway calls the retrieval service, which generates candidates: it looks up the user's embedding from the parameter-server fleet, runs an approximate-nearest-neighbor query against the sharded index, and returns a few hundred candidate item identifiers from a catalog of billions. Those candidates carry only identifiers, so the next stage, the feature fetch, reads the dense and sparse features for the user and for every candidate from the online feature store in one batched call. The enriched candidates go to the ranking service, a fleet of accelerator-backed inference servers that scores each candidate with the deep model from Section 38.4 and returns scores. Finally the re-rank stage applies diversity, freshness, and business-policy rules, trims to the page size, and the gateway returns the list.

Each arrow in that path is a network hop between fleets that scale independently, and the request is only as fast as the slowest stage it must wait for. Retrieval queries the embedding fleet built in Chapter 11 and the ANN index built in Chapter 25; ranking runs on the distributed inference substrate of Chapter 23. The architecture does not invent these; it composes them, and composition is where the latency goes.

2. Latency Budgeting: Slicing the SLO Across Stages Intermediate

The serving loop runs in sequence: retrieval must finish before feature fetch can name the candidates, feature fetch before ranking can score them, ranking before re-rank can reorder them. When stages run in series, their latencies add, so the end-to-end latency is the sum of the per-stage latencies,

and the architect's job is to hand each stage a budget $B_s$ whose sum leaves headroom under the SLO,

The slack term $T_{\text{slack}}$ is not waste; it absorbs the gateway overhead, the network hops between fleets, serialization, and the small variance that every stage carries. Because the product cares about the tail, the budgets are written against a high percentile (p99), not the mean, which ties this section directly to the tail-latency analysis of Section 34.7: a stage whose mean is comfortably inside budget can still blow the SLO if its p99 is three times its mean, which is the normal state of affairs for a stage that fans out to many shards and waits for the slowest. A budget is a promise about a percentile, and the percentile is where fan-out punishes you.

Budgets also bound the optimization you are allowed to do. The ranking model can be made more accurate by scoring more candidates or by using a larger network, but only up to the point where its p99 still fits $B_{\text{rank}}$. This is the lever that connects architecture to model design: the per-node efficiency techniques of Chapter 22 (quantization, batching, KV-cache discipline) exist precisely to buy a fatter model inside a fixed time slice. The budget is the contract; per-node efficiency is how you afford a better model under it.

3. Service Topology and Capacity from the Request Rate Intermediate

Latency sizes the depth of the pipeline; throughput sizes its width. Given a peak request rate of $\lambda$ requests per second and a stage whose service time is $T_s$ seconds, the average number of requests in flight at that stage follows Little's law,

and the number of replicas the fleet needs is the request rate divided by what one replica can sustain, with a safety factor for the tail and for failures,

The stages have wildly different $\mu_s$. A gateway replica forwards thousands of requests per second; an ANN shard serves a few thousand; a GPU ranking replica, scoring hundreds of candidates through a deep network, serves only a few hundred. So the ranking fleet is by far the widest and most expensive, and the total cost is the sum over fleets of replica count times unit price,

which is dominated by the accelerator-backed ranking fleet. This is why so much of the architecture's economics live in the ranking stage, and why Chapter 33 scheduling and bin-packing of those GPU replicas matters: a 10% improvement in ranking throughput per replica is a 10% cut in the largest line of the bill. Autoscaling tracks $\lambda$ over the day, adding replicas before the morning peak and retiring them at night, while a cache in front of the heaviest stages (popular users, popular candidate sets) lowers the effective $\lambda$ that reaches them.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: ranking-fleet }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: ranking }
  minReplicas: 40
  maxReplicas: 200                 # ranking is the widest, most expensive fleet
  metrics:
    - type: Pods                   # scale on p99 latency, not CPU, to defend the SLO
      pods:
        metric: { name: inference_p99_ms }
        target: { type: AverageValue, averageValue: "40" }

4. Two Loops: Offline Training and Online Serving Intermediate

The orange path in Figure 38.8.1 is a second, much slower loop that the request path never blocks on. Serving emits a log of impressions and clicks; streaming pipelines (the online AI of Chapter 9) turn those events into training features and write the freshest of them straight into the online feature store so that serving sees up-to-date signals within seconds. On a slower cadence, a training fleet retrains the embedding tables and the ranking model, the model registry validates the candidate against held-out metrics and runs a canary against a slice of live traffic, and only then are the new artifacts (the index, the embedding weights, the ranker checkpoint) published into the live services. That publish step is a hot swap: the serving fleets load the new artifact behind a version flag and cut over without dropping requests, which is exactly the model-deployment discipline of Chapter 26 applied to a recommender.

Keeping the two loops separate is what lets the system be both fresh and fast. The online loop is optimized for tail latency and never waits on training; the offline loop is optimized for model quality and throughput and never touches the request path. The only coupling is the artifact handoff and the log stream, and both are asynchronous. When the loops are accidentally coupled, for example when a feature is computed live from a database the training job also hammers, the architecture loses its independence and a training-side spike leaks into serving latency. The separation is a design rule, not an accident.

5. Reliability: Graceful Degradation Under the SLO Advanced

At the scale of five fleets and hundreds of replicas, something is always partly broken: a ranking replica is restarting, an ANN shard is slow, the feature store is shedding load. The architecture answers the SLO not by assuming everything works but by deciding, per stage, what to return when it does not. The reliability engineering of Chapter 35 gives the vocabulary: timeouts, retries with budget, circuit breakers, and fallbacks. Each stage carries a hard deadline equal to its budget $B_s$; if the stage does not answer by its deadline, the request does not wait, it degrades. If ranking times out, return the candidates in retrieval order (a worse but valid list) rather than nothing. If the feature store times out, score with cached or default features. If retrieval times out, serve a precomputed popular-items list for the user's segment. A degraded answer inside the SLO beats a perfect answer that arrives after the slot is gone.

This fallback ladder is what makes the additive latency model of Section 2 safe in practice: no single slow stage can drag the request past the SLO, because each stage is capped at its budget and a miss converts to a fallback rather than to unbounded waiting. The tail-latency techniques of Section 34.7 (hedged requests, tied requests, sending a duplicate to a second replica and taking the first answer) attack the same problem from the other side, shrinking the tail so the fallback fires rarely. Together, budgets plus fallbacks plus hedging turn a crowd of independently-failing fleets into a service with a latency contract it actually keeps.

6. The Budget in Numbers Intermediate

The runnable demo below makes Sections 2, 3, and 5 concrete. It takes an SLO of 120 ms and a peak rate of 50,000 requests per second, assigns each of four stages a budget $B_s$ and a measured p99 $T_s$, and checks the additive constraint $\sum_s T_s \le T_{\text{SLO}}$ from Section 2. It then sizes each fleet with $R_s = \lceil \lambda / \mu_s \rceil$ and reports the in-flight count $L_s = \lambda T_s$ from Section 3, exposing the ranking fleet as the widest. Finally it degrades the ranking stage from 40 ms to 70 ms (a cold cache, a heavier model) and shows the same additive sum now blows the SLO, which is exactly the condition the fallback ladder of Section 5 must catch.

import math

SLO_MS = 120.0          # end-to-end p99 latency target
QPS    = 50_000         # peak requests per second the system must absorb

# (stage, budget_ms, measured_p99_ms, per_replica_throughput_qps)
stages = [
    ("gateway + re-rank",  10.0,  8.0, 5000),
    ("retrieval (ANN)",    35.0, 31.0, 1200),
    ("feature fetch",      25.0, 22.0, 2500),
    ("ranking inference",  45.0, 40.0,  400),
]

budget_total   = sum(b for _, b, _, _ in stages)
measured_total = sum(m for _, _, m, _ in stages)

print(f"SLO (end-to-end p99)      : {SLO_MS:.0f} ms")
print(f"sum of stage budgets      : {budget_total:.0f} ms   (slack {SLO_MS - budget_total:+.0f} ms)")
print(f"measured end-to-end p99   : {measured_total:.0f} ms   -> "
      f"{'WITHIN' if measured_total <= SLO_MS else 'BLOWS'} SLO")
print()
print(f"{'stage':<20}{'budget':>8}{'meas':>7}{'in-flight':>11}{'replicas':>10}")
print("-" * 56)
for name, b, m, cap in stages:
    in_flight = QPS * (m / 1000.0)            # Little's law: L = lambda * W
    replicas  = math.ceil(QPS / cap)          # R = ceil(lambda / mu)
    print(f"{name:<20}{b:>6.0f}ms{m:>5.0f}ms{in_flight:>11.0f}{replicas:>10}")

total_replicas = sum(math.ceil(QPS / cap) for _, _, _, cap in stages)
print("-" * 56)
print(f"{'total fleet':<20}{'':>13}{'':>11}{total_replicas:>10}")
print()

# One stage regresses: ranking p99 drifts 40 -> 70 ms (cold cache, larger model)
deg = [(n, b, (70.0 if n == 'ranking inference' else m), c) for n, b, m, c in stages]
deg_total = sum(m for _, _, m, _ in deg)
print("Ranking p99 drifts 40 -> 70 ms:")
print(f"  new end-to-end p99      : {deg_total:.0f} ms   -> "
      f"{'WITHIN' if deg_total <= SLO_MS else 'BLOWS'} SLO by {deg_total - SLO_MS:+.0f} ms")

SLO (end-to-end p99)      : 120 ms
sum of stage budgets      : 115 ms   (slack +5 ms)
measured end-to-end p99   : 101 ms   -> WITHIN SLO

stage                 budget   meas  in-flight  replicas
--------------------------------------------------------
gateway + re-rank       10ms    8ms        400        10
retrieval (ANN)         35ms   31ms       1550        42
feature fetch           25ms   22ms       1100        20
ranking inference       45ms   40ms       2000       125
--------------------------------------------------------
total fleet                                        197

Ranking p99 drifts 40 -> 70 ms:
  new end-to-end p99      : 131 ms   -> BLOWS SLO by +11 ms

7. Reading the Whole System as One Picture Beginner

Step back from Figure 38.8.1 and the recommender is legible as a single object: a fast loop and a slow loop sharing two thin connections, a log stream down and an artifact handoff up. The fast loop spends a fixed budget across a fixed sequence of fleets and degrades rather than waits; the slow loop turns the fast loop's exhaust (its logs) into the fast loop's fuel (its models) on a cadence the fast loop never feels. Every component is something this chapter already built, and every cross-chapter dependency, the collectives of Chapter 4 inside training, the parameter servers of Chapter 11, the retrieval of Chapter 25, the serving of Chapters 23 and 24, the scheduling of Chapter 33, the reliability of Chapter 35, is here as a service with a budget and a fallback. The final section hands this architecture back to you as a buildable project, growing a single-machine baseline into the full federation of services one stage at a time, with a measurable milestone at each step. That continues in Section 38.9.

8. The Generative Turn: From Cascades to Autoregressive Ranking Advanced

The retrieve-then-rank funnel described across Sections 38.3 and 38.4 has driven production recommenders for a decade. From the distributed-systems angle, its strength is decomposability: each stage is an independent fleet, can be scaled and optimized separately, and a straightforward latency budget governs the whole pipeline. Yet the cascade carries three structural weaknesses that have become visible only as GPU cluster budgets grew large enough to challenge them. Understanding those weaknesses motivates the generative recommender architectures, such as TIGER and HSTU, that began displacing or augmenting the cascade in production between 2024 and 2026.

The first weakness is the hard candidate ceiling: the retrieval stage narrows billions of items to a few thousand, and any item outside that shortlist cannot be ranked, no matter how good the ranking model is. The ceiling is not a tuning parameter; it is a consequence of the latency budget ($B_{\text{retrieve}} \approx 30\text{--}35$ ms in the example above) and the cost of an ANN query. The second weakness is the absence of cross-item interactions at retrieval time. An approximate-nearest-neighbor lookup scores each item against the user vector independently; it cannot model "given that we are already going to show item A, how valuable is item B?" The third weakness is the model separation: the retrieval model (a two-tower embedding similarity) and the ranking model (a deep feature-interaction network) are trained with different objectives, on different data, and optimised at different points in time, so the two stages are never jointly optimal. The cascade was the right architecture when compute was the binding constraint; today the binding constraint is often the hard ceiling and the lack of joint optimisation.

TIGER: Autoregressive Generation of Semantic Item IDs

TIGER (Transformer Index for GEnerative Recommenders, arXiv:2305.05065) reframes recommendation as a sequence-to-sequence generation problem. The key step is the assignment of semantic IDs to items. Each item's content embedding is quantised by hierarchical clustering into a short tuple of integer codes $(c_1, c_2, \ldots, c_D)$ where each $c_i \in \{0, 1, \ldots, K-1\}$ and $K$ is the codebook size. Two items that are semantically similar end up with IDs that share a long prefix, which gives the model an inductive bias the two-tower retrieval stage entirely lacks. The vocabulary of all possible item IDs has size

so for $K = 256$ and $D = 4$ the vocabulary is $256^4 = 4{,}294{,}967{,}296$, roughly four billion distinct IDs, large enough to cover any catalog without ID collisions. A sequence Transformer then takes the user's interaction history and generates the next item's semantic ID token-by-token, conditioning each code $c_i$ on the codes $c_1, \ldots, c_{i-1}$ already generated,

This collapses retrieval and ranking into a single forward pass through one model. There is no separate ANN index, no two-tower similarity, and no hard candidate ceiling: the model generates any item in the catalog, constrained only by the semantic ID vocabulary. Retrieval and ranking are jointly optimised end-to-end on the same loss.

HSTU: NLP-Style Scaling Laws for Recommendation

Meta's HSTU (Hierarchical Sequential Transduction Units, arXiv:2402.17152) takes a different but complementary step. Rather than replacing the cascade with pure generation, HSTU introduces a new sequence-modelling architecture designed specifically for the recommendation setting, where the "vocabulary" is a user's interaction history over items (rather than words) and the history can be extremely long. The paper's central empirical finding is that HSTU obeys NLP-style scaling laws: ranking quality improves predictably and smoothly as the model grows, with no plateau visible up to roughly 1.5 trillion parameters in Meta's production deployment. This is the first time recommendation ranking has shown the same scaling behaviour as language modelling, and it validates the case for investing GPU cluster budget in a single large ranking model rather than in a cascade of smaller specialist models.

From the distributed-systems angle, HSTU is still a ranking model rather than a pure generative retriever; it operates on the survivors of a retrieval pass. Its significance is that it replaces the DLRM-style cascade (the architecture that Section 38.4 builds on) with a single large transformer whose serving cost scales predictably with size, and whose quality gains from compute are guaranteed rather than speculative. The model also introduces new distributed serving requirements: its large size demands tensor parallelism across multiple accelerators per replica, and its sequential attention over long user histories requires careful KV-cache management of the kind described in Chapter 24.

Distributed Serving Implications

The cascade and the generative recommender differ fundamentally in their serving topology, and the difference reshapes the capacity calculation of Section 3. In the cascade, retrieval and ranking are feed-forward: given a fixed set of candidates, scoring is one matrix multiplication per candidate per layer, and the entire set can be scored in one batched forward pass. In a generative recommender, the next item is produced by autoregressive decoding: the model generates one code token per step, and each step requires a separate forward pass conditioned on the previously generated tokens. For a semantic ID of depth $D$, producing one item recommendation requires $D$ forward passes rather than one, and producing a list of $N$ diverse recommendations via beam search with beam width $B$ requires $D \cdot B$ forward passes with a reranking step at the end.

This autoregressive serving pattern is structurally identical to the LLM serving problem described in Chapter 24. The same techniques apply: tensor parallelism to split the model across accelerators when it is too large for one, KV-cache management to avoid recomputing attention over the already-generated prefix, and speculative decoding to predict the next code token with a smaller draft model and verify with the full model in one batched pass. Speculative decoding is particularly attractive for semantic IDs because the hierarchical codebook structure means code tokens at depth $d+1$ are strongly predicted by depth $d$: a small draft model that knows the first code in a semantic ID tuple can predict the second with high accuracy, and the verification step is free when the prediction is right.

Tradeoffs: When the Cascade Stays

The generative turn does not make the cascade obsolete; it shifts the choice from "always cascade" to a deliberate tradeoff. Generative recommenders carry three costs the cascade avoids. First, autoregressive serving is slower: $D$ serial forward passes versus one batched forward pass, which may exceed the latency budget on latency-critical surfaces (live notifications, real-time ads auctions). Second, semantic ID tokenisation requires infrastructure to assign and maintain hierarchical codes for every item in the catalog, re-cluster when the catalog changes significantly, and handle new items before they have been assigned a stable semantic ID (the cold-start problem is harder in the generative setting). Third, context length grows with history: the sequence transformer must attend over the user's full interaction history, and a user with thousands of interactions requires a long context window and the KV-cache pressure that comes with it, exactly the tradeoffs of long-context LLM serving from Chapter 24.

The practical outcome is a hybrid architecture: a generative retriever (TIGER-style) or a large HSTU-style ranker on the main serving path where latency allows, and the traditional cascade as a fast fallback for latency-critical sub-paths and for cold-start items. The cascade remains the right choice wherever the hard candidate ceiling does not bind (small catalogs), wherever the latency budget is tight enough to rule out multiple autoregressive decoding steps, and wherever the semantic ID infrastructure has not yet been deployed.