"My accuracy on the validation set was excellent. It is just that the validation set and the world stopped being the same thing about three weeks ago, and nobody told either of us."
A Model That Trained Once and Believed Forever
A deployed model is a photograph of a world that keeps moving, so the central question of production AI is not "is the model accurate?" but "is the model still accurate, and how would we know in time?" The world that generated the training data drifts: user behavior shifts, an upstream feature pipeline changes units, a competitor launches, a season turns. The model does not announce its own decay; accuracy slips quietly while every system stays green. This final section of the chapter builds the instrument that catches that decay on a live stream, by watching the distribution of inputs and predictions window by window and raising an alert the moment a statistic crosses a line. Because the model runs not on one server but across a fleet of replicas, the instrument must itself be distributed: each replica measures locally, and a coordinator merges those measurements into one cluster-wide verdict using the same mergeable sketches we built for distributed aggregation in Section 6.8. With monitoring in place, Part II is complete: we will have carried data from raw bytes all the way to a model that watches its own freshness in production.
Every chapter so far has moved data toward a model: partitioned it (Chapter 6), reshaped it in DataFrames (Chapter 7), loaded it from distributed storage (Chapter 8), and turned it into an unbounded stream of events that feeds online features and real-time predictions (Sections 9.1 through 9.8). That pipeline assumed something it never checked: that the live data still resembles the data the model learned from. When that assumption fails, the pipeline keeps running flawlessly and the answers quietly get worse. The job of this section is to make that silent failure loud.
1. Two Ways a Model Goes Stale Beginner
A model learns a relationship between inputs $x$ and a target $y$, which probability gives us the joint distribution $P(x, y) = P(y \mid x)\, P(x)$. Staleness is any change in that joint distribution after deployment, and it comes in two flavors that demand different responses. The first is data drift (also called covariate drift): the input distribution $P(x)$ shifts while the underlying rule $P(y \mid x)$ stays fixed. New kinds of traffic arrive, a feature's range moves, a sensor recalibrates. The model's learned mapping is still correct, but it is now being asked questions from a region of input space it saw rarely during training, so its answers there are less reliable. The second is concept drift: the rule itself, $P(y \mid x)$, changes. The same input now implies a different outcome. A transaction pattern that meant "legitimate" last year means "fraud" today; a search query that meant one intent now means another. Concept drift is the more dangerous kind, because retraining on fresh inputs alone will not fix it; you need fresh labels that reflect the new rule.
The practical trouble is that the two often look identical from where we stand. In production we usually see the inputs $x$ and the model's predictions immediately, but the true labels $y$ arrive late or not at all: a loan's default status is known months later, a recommendation's true value is never directly observed. So our earliest warning has to come from what we can see right away, the distributions of features and predictions, and only later can delayed labels confirm whether accuracy actually fell. This is why drift monitoring is layered: fast, label-free distribution checks raise a flag, and slower, label-based metric checks adjudicate it.
If true labels arrived instantly, monitoring would be trivial: track accuracy and alert when it drops. They almost never do. The defining constraint of production monitoring is the label delay, so the first line of defense watches the things available with zero delay, the input features and the model's own output distribution, and treats a shift in those as a leading indicator of a coming accuracy drop. Distribution monitoring does not prove the model got worse; it proves the model is now operating in conditions it was not validated for, which is a reason to investigate before the damage shows up in a delayed metric.
2. Measuring Drift on a Window Intermediate
To detect a shift we need a number that compares a fixed reference distribution, captured at deployment from data we trust, against a live distribution measured over a recent window of the stream (windows are exactly the bounded slices of an unbounded stream we defined in Section 9.2). Two comparisons dominate practice. The first is the Population Stability Index (PSI). Bin both distributions into the same $B$ buckets, let $p_b$ be the reference fraction in bucket $b$ and $q_b$ the live fraction, and define
$$\mathrm{PSI} = \sum_{b=1}^{B} (q_b - p_b)\,\ln\frac{q_b}{p_b}.$$PSI is a symmetrized relative-entropy-style score: it is zero when the two histograms match and grows as they diverge. A widely used convention reads $\mathrm{PSI} < 0.1$ as stable, $0.1 \le \mathrm{PSI} < 0.25$ as a moderate shift worth watching, and $\mathrm{PSI} \ge 0.25$ as a material shift that should raise an alert. The second comparison is the two-sample Kolmogorov-Smirnov (KS) test, which needs no binning: it takes the largest gap between the two empirical cumulative distribution functions, $D = \sup_x |F_{\text{ref}}(x) - F_{\text{live}}(x)|$, and turns it into a p-value for the hypothesis that both samples came from the same distribution. PSI is cheap, interpretable, and the standard in risk and credit settings; the KS test is distribution-free and good for continuous features where you would rather not choose bins. You apply the same idea to the model's prediction distribution, which catches drift even when no single input feature moved much but the model's overall behavior shifted.
The code below builds a PSI drift detector over a stream whose generating distribution shifts halfway through. It fixes decile bins from a reference sample, then walks fixed-size windows, computes PSI for each, and flags the window the moment the score crosses the alert line.
import random, math
random.seed(11)
N_REF, WIN, N_WINDOWS, SHIFT_AT, ALERT, EPS = 20_000, 2_000, 12, 6, 0.25, 1e-6
ref = [random.gauss(0.0, 1.0) for _ in range(N_REF)] # calibration sample
ref_sorted, n_bins = sorted(ref), 10
edges = [-math.inf] + [ref_sorted[k * N_REF // n_bins] for k in range(1, n_bins)] + [math.inf]
def histogram(values): # bin into fixed reference edges
counts = [0] * n_bins
for v in values:
for b in range(n_bins):
if edges[b] <= v < edges[b + 1]:
counts[b] += 1; break
total = sum(counts)
return [max(c / total, EPS) for c in counts] # EPS floors empty bins
p = histogram(ref) # reference probability vector
def psi(q): return sum((q[b] - p[b]) * math.log(q[b] / p[b]) for b in range(n_bins))
print(f"{'window':>6} | {'gen mean':>8} | {'PSI':>7} | status")
print("-" * 42)
for w in range(N_WINDOWS):
mu = 0.0 if w < SHIFT_AT else 0.8 # distribution shifts at window 6
q = histogram([random.gauss(mu, 1.0) for _ in range(WIN)])
score = psi(q)
status = "ALERT" if score >= ALERT else ("watch" if score >= 0.1 else "ok")
print(f"{w:>6} | {mu:>8.2f} | {score:>7.4f} | {status}")
p is fixed once from a trusted calibration sample; each live window is binned into the same edges and scored against it, and the score is compared to the alert threshold.window | gen mean | PSI | status
------------------------------------------
0 | 0.00 | 0.0040 | ok
1 | 0.00 | 0.0073 | ok
2 | 0.00 | 0.0019 | ok
3 | 0.00 | 0.0020 | ok
4 | 0.00 | 0.0029 | ok
5 | 0.00 | 0.0036 | ok
6 | 0.80 | 0.6445 | ALERT
7 | 0.80 | 0.6117 | ALERT
8 | 0.80 | 0.5792 | ALERT
9 | 0.80 | 0.5572 | ALERT
10 | 0.80 | 0.6023 | ALERT
11 | 0.80 | 0.5400 | ALERT
The detector did exactly one useful thing, and it did it without a single label: it converted a change in the world into a number that crossed a threshold within one window of the change. The diagram in Figure 9.9.1 shows the shape of that signal and, on its right half, how the same statistic is assembled from many serving replicas rather than one process.
The most expensive drift incidents share a signature: every infrastructure dashboard stayed green. CPU was fine, latency was fine, error rate was zero, uptime was a row of nines. The model served every request successfully and confidently, and was wrong more and more often. Liveness monitoring asks "is the service up?" Drift monitoring asks "is the service right?" Those are different questions, and only one of them has a green light that can lie to you.
3. Monitoring Across a Fleet of Replicas Intermediate
Code 9.9.1 measured one stream in one process. A real serving system runs the model on dozens or hundreds of replicas behind a load balancer (the real-time inference fleet of Section 9.8), and the drift question is about the aggregate traffic, not any one replica. The naive approach, shipping every raw prediction and feature vector to a central place to be re-binned, recreates exactly the all-data-to-one-node bottleneck this book keeps teaching you to avoid. The right move is the one from distributed aggregation: each replica maintains a small mergeable sketch of its local traffic, and the central monitor combines the sketches.
For PSI the sketch is simply a per-bin count vector; merging is element-wise addition, which is associative and commutative, so replicas can be combined in any order and partial merges along a tree give the same total as a flat sum. This is the same mergeability property that made distributed counting, distinct-count (HyperLogLog), and quantile sketches work in Section 6.8; a histogram is the most basic mergeable sketch there is. Each replica sends a vector of $B$ integers per window instead of thousands of raw records, the monitor sums them into one cluster-wide histogram, normalizes, and scores it against the reference. The bytes on the wire drop by orders of magnitude and the merged score is exact, not an approximation, because summing counts loses nothing.
The same pattern carries the metrics that arrive with delayed labels. When ground truth finally lands, each replica (or a downstream join job) accumulates a small confusion-matrix sketch, and the monitor merges those into a fleet-wide accuracy, precision, or calibration curve, then watches that across windows as the slower, definitive confirmation of what the distribution alert predicted. Two layers, same mechanics: cheap label-free distribution sketches for early warning, merged label-based metric sketches for confirmation, both built from per-replica summaries that add.
Code 9.9.1 spelled out PSI from histograms to teach the mechanics. In production you reach for a maintained library. Evidently computes PSI, the KS test, Wasserstein distance, and prediction-drift reports over reference-versus-current frames in a few lines and renders a dashboard. NannyML goes further and estimates model performance under covariate drift before labels arrive, which is precisely the label-delay problem of Section 1. river targets online streams directly with incremental drift detectors such as ADWIN and the Page-Hinkley test that update one event at a time:
# pip install evidently river
from river import drift
detector = drift.ADWIN() # adaptive windowing; no fixed window size
for x in feature_stream: # one event at a time, O(1) memory amortized
detector.update(x)
if detector.drift_detected:
trigger_retrain() # ADWIN found a change point in the stream
river. ADWIN maintains an adaptive window and flags change points automatically, replacing the fixed-window PSI loop of Code 9.9.1; the library handles windowing, the statistical test, and the memory bound that a from-scratch detector would have to manage by hand.Who: An ML platform engineer responsible for a fraud-scoring service running across forty inference replicas.
Situation: The model had held steady at high precision for months, and the team had stopped looking at it closely.
Problem: An upstream team changed a transaction-amount field from cents to dollars, dividing one feature by a hundred across the whole stream, and shipped it without a heads-up.
Dilemma: True fraud labels lag by weeks, so waiting for an accuracy drop meant weeks of degraded scoring; but feature-level alerts can be noisy, and nobody wanted a pager that cried wolf.
Decision: They trusted the distribution layer: a per-replica PSI sketch on each feature, merged centrally per five-minute window, with an alert at $\mathrm{PSI} \ge 0.25$.
How: The amount feature's merged histogram collapsed into the lowest bins; its PSI shot past the line within two windows while every other feature stayed flat, pointing straight at the culprit.
Result: The alert fired the same afternoon as the deploy, the units bug was reverted before a single fraud label came back, and no accuracy loss ever materialized.
Lesson: Distribution monitoring catches upstream data-contract breaks that no liveness check would ever see, and isolating drift per feature turns a vague "something is off" into a named root cause.
4. From Alert to Action: Triggering Retraining Advanced
An alert is only useful if it drives an action, and drift has three standard responses ordered by cost. The cheapest, when the shift is covariate and the rule still holds, is to do nothing but watch; not every distribution wobble warrants a retrain, and over-reacting churns models for no accuracy gain. The middle response is an online update: keep learning incrementally from the fresh stream so the model tracks a slowly moving target, which is the natural fit when drift is gradual and some labels trickle in. The heaviest response is a full retrain on a refreshed dataset, warranted when the alert is sharp and sustained, as in Output 9.9.1, or when delayed labels confirm a real accuracy drop. A mature pipeline wires the merged alert directly into the retraining trigger so the loop closes automatically: detect, validate against a holdout, and promote the new model only if it beats the incumbent.
The machinery for those updates is the subject of the next part of the book. Online and incremental optimization, mini-batch and streaming SGD, and the question of how to update a model from a moving data distribution without destabilizing it are exactly what Chapter 10 opens with. The full production loop, where drift detection, automated retraining, validation, canary rollout, and rollback live together as a governed system, is the domain of distributed MLOps in Chapter 26. This section builds the sensor; those chapters build the control system the sensor feeds.
The sharpest open problem in monitoring is the label delay itself: distribution drift tells you conditions changed, but not by how much accuracy actually fell. A growing line of work estimates model performance directly under covariate shift without waiting for labels. Confidence-based estimators such as NannyML's CBPE (Confidence-Based Performance Estimation) and DLE reweight the model's own predicted probabilities by the drifted input density to project accuracy forward; importance-weighting and density-ratio methods give error bounds under covariate-shift assumptions. In parallel, recent surveys of unsupervised concept-drift detection (Bayram et al., 2022 onward, with active 2024 to 2026 follow-ups) push toward detectors that separate covariate drift from genuine $P(y \mid x)$ drift using only inputs and predictions, and toward calibration-drift monitors that watch whether predicted probabilities stay honest. We arrive at these methods with the right instruments after the optimization and MLOps machinery of Parts III and V; for now, note that the field is racing to close the gap between "the inputs moved" and "and here is what it cost you."
5. Chapter 9 in One View, and the End of Part II Beginner
This section closes Chapter 9, and Chapter 9 closes Part II. It is worth seeing the whole arc at once. Chapter 9 took the static datasets of the earlier chapters and set them in motion. We started by contrasting batch processing, which sees a complete bounded dataset, with stream processing, which must produce answers over an unbounded, never-finished sequence of events (Section 9.1). We learned to carve that stream into windows so that aggregation has something finite to operate on (Section 9.2), and to separate event time (when something happened) from processing time (when we saw it), because the two diverge whenever events arrive out of order (Section 9.3). Watermarks gave us a principled way to decide when a window is complete enough to emit despite late events (Section 9.4). The Kafka-style distributed log gave the stream a durable, replayable, partitioned backbone that many producers and consumers can share (Section 9.5), and Spark Structured Streaming and Flink gave us the engines that run windowed computations over it at scale (Section 9.6). On that foundation we computed online features (Section 9.7), served real-time predictions from a distributed inference pipeline (Section 9.8), and finally, in this section, watched the deployed model for drift so it does not rot in place.
Part II is the data substrate of the whole book. Chapter 6 gave us MapReduce and the shuffle, the first model for computing over data too big for one machine. Chapter 7 raised that to Spark and distributed DataFrames. Chapter 8 handled distributed storage and the data-loading path that feeds training. Chapter 9 set the data in motion as streams, with windows, event time, watermarks, the Kafka log, stream engines, online features, real-time inference, and drift monitoring. Across all four chapters one idea recurs: split the data across machines, summarize each shard with something that merges, and recombine the summaries correctly, whether the summary is a MapReduce partial aggregate, a Spark shuffle partition, or a per-replica drift sketch. The merge that closed this chapter is a direct descendant of the all-reduce that opened the book in Section 1.1, and that lineage is the spine the rest of the book climbs.
Part II made one half of the thesis concrete: when the data outgrows one machine, you distribute it, and you summarize each shard with a mergeable operation so the pieces recombine into a correct whole. Every tool in this part, the MapReduce shuffle, the Spark DataFrame, the streaming window, the per-replica drift sketch, is an instance of that single move. But we have been distributing the data path while keeping the model as a thing trained on one box. Part III breaks that last assumption. The same logic that let eight workers compute one exact gradient in Section 1.1 now becomes the engine of training itself: distribute the optimization, merge the gradients, and keep many machines learning as one. The drift alert that ends Part II is, fittingly, the trigger that starts Part III's machinery running.
6. Project Ideas Intermediate
These are open-ended builds that exercise the whole chapter, not just this section. Each is sized for a weekend and grows naturally toward the distributed-training material of Part III.
Take Code 9.9.1 from a single loop to a real stream. Publish a synthetic feature stream into a Kafka topic (Section 9.5) whose distribution shifts at a known offset, consume it in tumbling windows with Spark Structured Streaming or Flink (Section 9.6), and compute a per-window PSI or KS statistic against a fixed reference. Emit an alert event to a second topic when the statistic crosses the threshold, and measure the detection latency: how many windows after the true shift did the alert fire? Then add a second, gradually drifting feature and show that a fixed threshold that catches the sharp shift either misses or lags the gradual one, motivating an adaptive detector like ADWIN from Code 9.9.2.
Simulate $R$ serving replicas, each consuming a partition of the stream and maintaining a local per-bin count sketch per window. Implement a central monitor that merges the $R$ sketches by element-wise addition into one cluster-wide histogram and scores it. Verify empirically that the merged PSI equals the PSI you would get by pooling all raw events centrally (it should, exactly), then measure the bytes-on-the-wire saving of shipping sketches versus raw records. For extra credit, route a distribution shift to only a subset of replicas (a bad deploy on part of the fleet) and confirm the merged score still catches it while a single-replica view might miss it.
Wire a drift alert to an action. Train a simple online model (logistic regression or a streaming tree) on the early, stable part of a stream, serve predictions, and let your detector from Project A watch the prediction distribution. When the alert fires, trigger an incremental update or a full retrain on the recent window, promote the new model only if it beats the incumbent on a held-out slice, and plot served accuracy over time with and without the retraining loop. Report the accuracy recovered and the cost paid (retrains triggered, including any false alarms), and discuss where on the do-nothing / online-update / full-retrain spectrum of Section 4 your data falls.
7. Exercises Intermediate
For each scenario, classify the staleness as data (covariate) drift, concept drift, or both, and state whether retraining on fresh inputs alone would fix it: (a) a demand-forecasting model after a new product category launches that it never saw in training; (b) a spam classifier after spammers deliberately rewrite their messages to evade it while the words themselves look ordinary; (c) a sensor-fault detector after a firmware update halves the sampling rate of every sensor. For each, name which signal you would watch first given that true labels are delayed, and explain why a label-free distribution check would or would not raise an alert.
Extend Code 9.9.1 to simulate three replicas. Split each window's events across the three replicas at random, have each build its own per-bin count vector, and merge the three by element-wise addition before normalizing and computing PSI. Compare the merged PSI to the single-process PSI from Output 9.9.1 on the identical event set. Confirm they match to floating-point rounding, then explain in one paragraph which property of the histogram sketch (and which property of the PSI formula) makes the distributed computation exact rather than approximate, referencing the mergeable sketches of Section 6.8.
Using the detector from Code 9.9.1, sweep the window size (for example 250, 1000, 4000 events) and the alert threshold (for example 0.1, 0.25, 0.5) and, for a fixed shift, record two quantities for each setting: the detection latency in events after the true shift, and the number of false alerts during the stable period. Plot the trade-off. Argue from your numbers why a smaller window detects faster but alerts more noisily, and connect this directly to the watermark lateness trade-off of Section 9.4, where waiting longer also bought completeness at the cost of latency.