Distributed Drift Detection

"Every replica swore its little slice of traffic looked completely normal. It took all of us, added up, to notice the world had moved on without us."

A Histogram That Only Saw One Shard

The previous section gave us fleet-wide observability: latency, throughput, and error rates aggregated across every replica so that the serving system reports its own health as one number rather than a thousand. Those metrics tell you whether the system is running. They say nothing about whether the model is still right. A model can be fast, available, and quietly wrong, returning a prediction in eight milliseconds for an input that belongs to a world it was never trained on. Drift detection is the missing half of monitoring: instead of watching the system's vital signs, it watches the statistical relationship between the data the model sees today and the data it learned from, and raises an alarm when that relationship decays. It is the sensor that decides when the continuous-training pipeline of Section 26.4 should fire.

This deepens the streaming view from Chapter 9, where drift appeared as a property of a single online learner reacting to a changing stream. Here the model is fixed, the stream is the aggregate of every replica's traffic, and the question is not "how do I adapt one learner online?" but "how do I detect, across a whole serving fleet, that the deployed model has gone stale, and trigger a retrain before users notice?"

1. Three Kinds of Drift, and the Label-Delay Problem Beginner

Drift is not one phenomenon. It helps to keep three kinds separate, because they have different causes, different detectability, and different remedies. Data drift (also called covariate drift) is a shift in the input distribution: $P(x)$ changes while the underlying input-to-output rule stays fixed. A fraud model starts seeing transactions from a new country; a vision model starts seeing photos from a new phone's camera. Concept drift is a shift in the relationship itself: $P(y \mid x)$ changes, so the same input should now map to a different answer. Spending patterns that were benign last year are fraudulent this year; the meaning of a word shifts. Prediction drift is a shift in the model's output distribution $P(\hat{y})$: the mix of classes the model emits, or the shape of its score histogram, moves. Prediction drift is often the first observable symptom of the other two, because it is computed purely from outputs you already have.

The practical consequence is a layered strategy. You monitor the cheap, immediate, unlabeled signals continuously (input distributions, output distributions, confidence), and you reconcile them against true accuracy whenever the delayed labels finally land. A spike in data drift that is later confirmed by a drop in labeled accuracy calibrates your thresholds; a spike that is not confirmed teaches you that this particular input shift was harmless. Over time the proxies become trustworthy early sensors for a quantity you can only verify in arrears.

2. Detection Methods: Measuring the Distance Between Two Distributions Intermediate

Every unlabeled drift detector reduces to the same shape: summarize a fixed reference window (the distribution the model was validated on), summarize a sliding current window of live traffic, and compute a distance between the two. When the distance crosses a threshold, you declare drift. The detectors differ only in what feature they summarize and which distance they use.

For a single numeric feature, the workhorse is the Population Stability Index (PSI), which bins both windows onto a shared grid and sums the per-bin divergence. With reference proportions $r_b$ and current proportions $c_b$ over bins $b = 1, \dots, B$,

This is a symmetrized, binned cousin of the Kullback-Leibler divergence. A common rule of thumb reads $\mathrm{PSI} < 0.1$ as no material shift, $0.1 \le \mathrm{PSI} < 0.2$ as moderate, and $\mathrm{PSI} \ge 0.2$ as a population that has materially moved and warrants action. The Kolmogorov-Smirnov (KS) statistic instead takes the maximum gap between the two empirical cumulative distributions, $D = \sup_x |F_{\text{ref}}(x) - F_{\text{cur}}(x)|$, and comes with a hypothesis test. For high-dimensional features such as embeddings, the Maximum Mean Discrepancy (MMD) compares distributions in a kernel feature space without binning, which is what you reach for when the thing that drifted is a 768-dimensional vector rather than a scalar. Alongside these distribution tests, two almost-free signals deserve continuous monitoring: prediction confidence (a model growing systematically less certain is often the first whisper of drift) and class balance (the proportion of each predicted class), both of which are prediction-drift proxies computable from outputs you are already logging.

3. Why Drift Detection Is a Distributed-Aggregation Problem Intermediate

Here is the distributed twist that makes this a Part V problem rather than a textbook statistics exercise. The distribution you care about is the distribution of the whole traffic stream, across every replica in the serving fleet. No single replica sees it. A load balancer might route a new geography's traffic disproportionately to three replicas out of fifty; each of those three sees a local shift that looks like noise, while the fleet-wide shift that actually matters is invisible to all of them individually. Computing drift correctly means computing it over the union of every replica's traffic, which is exactly the aggregation pattern this book has used since Chapter 6.

The mechanism is a mergeable summary. Each replica does not ship its raw predictions to a central place; that would be a firehose. Instead each replica maintains a histogram (or a quantile sketch, the same family of mergeable structures introduced with MapReduce-style aggregation in Chapter 6) over a shared, fleet-wide set of bin edges. Because the bins are agreed in advance, the histograms are additive: the central aggregator simply sums the per-replica bin counts to obtain the exact fleet-wide histogram, then normalizes once. This rides the very same telemetry path that Section 26.6 built for latency and error metrics; drift sketches are just another mergeable metric flowing through the same pipeline. The aggregated current distribution is then compared against the stored reference, and a single fleet-wide drift score comes out.

4. A Fleet-Wide Drift Detector From Scratch Intermediate

The code below makes the whole pipeline concrete with nothing but NumPy. Four replicas each bin their own slice of traffic over shared edges; the aggregator sums the histograms into one fleet-wide distribution; a PSI score is computed against a reference window each monitoring window; and a retrain trigger fires when PSI crosses $0.2$. The stream is deliberately stable for four windows, then drifts: the input mean slides and the spread widens, the signature of covariate drift.

import numpy as np

rng = np.random.default_rng(7)

# Fixed bin edges agreed fleet-wide so every replica's histogram is mergeable.
EDGES = np.linspace(-6.0, 6.0, 21)          # 20 bins over the score range
N_REPLICAS = 4
WINDOW = 5_000                              # predictions per replica per window
THRESHOLD = 0.2                            # PSI > 0.2 == material drift, retrain

def replica_histogram(samples):
    """One replica bins ONLY its own slice of traffic into a shared grid."""
    counts, _ = np.histogram(samples, bins=EDGES)
    return counts.astype(np.float64)

def fleet_distribution(per_replica_counts):
    """Central aggregator sums mergeable histograms, then normalizes once."""
    total = np.sum(per_replica_counts, axis=0)
    return total / total.sum()

def psi(reference, current, eps=1e-6):
    """Population Stability Index between two binned distributions."""
    r = np.clip(reference, eps, None)
    c = np.clip(current, eps, None)
    return float(np.sum((c - r) * np.log(c / r)))

# Reference window: the input score distribution the model was validated on.
ref_counts = [replica_histogram(rng.normal(0.0, 1.0, WINDOW)) for _ in range(N_REPLICAS)]
reference = fleet_distribution(ref_counts)

# Stream eight monitoring windows. The serving population drifts after window 4:
# the mean of the input scores slides and the spread widens (covariate drift).
print(f"{'window':>6} {'fleet mean':>11} {'PSI':>8}  trigger")
print("=" * 38)
for w in range(8):
    if w < 4:
        mu, sigma = 0.0, 1.0                # stable regime
    else:
        mu, sigma = 0.6 + 0.25 * (w - 4), 1.0 + 0.18 * (w - 4)   # drifting regime
    per_replica = [replica_histogram(rng.normal(mu, sigma, WINDOW)) for _ in range(N_REPLICAS)]
    current = fleet_distribution(per_replica)
    score = psi(reference, current)
    fired = "RETRAIN" if score > THRESHOLD else "."
    print(f"{w:>6} {mu:>11.2f} {score:>8.4f}  {fired}")

window  fleet mean      PSI  trigger
======================================
     0        0.00   0.0011  .
     1        0.00   0.0008  .
     2        0.00   0.0018  .
     3        0.00   0.0008  .
     4        0.60   0.3682  RETRAIN
     5        0.85   0.6938  RETRAIN
     6        1.10   1.1481  RETRAIN
     7        1.35   1.6833  RETRAIN

The detector did exactly what Figure 26.7.1 promised: it stayed quiet while the fleet-wide distribution matched the reference, and it fired the moment that distribution moved, with a score that grows monotonically as the gap widens. Note that the PSI was computed on the summed histogram; had we instead averaged four per-replica PSI scores, a shift concentrated on a few replicas could have been diluted below the threshold, which is precisely the local-blindness trap that motivates fleet-wide aggregation.

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])    # per-feature tests, auto-chosen
report.run(reference_data=reference_df, current_data=current_df)
report.as_dict()["metrics"][0]["result"]["dataset_drift"]   # True / False fleet verdict

5. Drift in LLM Serving: Embeddings and Quality Proxies Advanced

For a classifier, the monitored feature is a tidy numeric score. For a large language model serving free-form text, there is no single scalar to bin, and the methods generalize rather than transfer directly. The standard move is to monitor the embedding distribution of inputs and outputs: encode each prompt and each response into a vector, and track that high-dimensional distribution over windows with a binning-free distance such as MMD, or by reducing to a few summary statistics (mean cosine distance to the reference centroid, the share of inputs whose nearest reference cluster is far away). A surge of out-of-distribution prompts, a new jailbreak template, a topic the model was never tuned for, shows up as the input embedding cloud drifting away from the reference cloud, long before any human flags a bad answer.

Because an LLM has no cheap ground-truth label either, output quality is tracked through quality proxies: response length distribution, refusal rate, toxicity and safety-classifier scores, retrieval-grounding scores in a RAG system, and increasingly an LLM-as-judge score sampled on a fraction of traffic. Each is a mergeable per-replica statistic that aggregates fleet-wide exactly like the histogram in Code 26.7.1, so the distributed machinery is unchanged; only the monitored quantity is richer.

6. From Detection to Action: Closing the Loop Without Crying Wolf Intermediate

A drift score is only useful if it drives an action, and the action is a small state machine. Below the threshold, do nothing. Above it, raise an alert to the on-call channel and, depending on confidence, trigger the retraining pipeline that Section 26.1 and Section 26.4 built, which pulls a fresh labeled window, retrains, evaluates against a held-out set, and promotes the new model only if it actually beats the incumbent. The detector closes the loop from a deployed-and-decaying model back to a freshly trained one, which is the whole point of an MLOps pipeline: a model that maintains itself.

The dominant failure mode is the false alarm, and two causes account for most of them. The first is seasonality: traffic legitimately shifts on weekends, holidays, and promotions, and a detector that treats every shift as decay will retrain on freak data and ship regressions, as the Black Friday note warned. The remedy is to compare against a seasonally matched reference (this Monday against prior Mondays, not against an all-time average) and to require persistence, a score that stays high for several consecutive windows rather than one spike. The second is multiple testing: with hundreds of features each tested every window, some will cross any fixed threshold by pure chance, and the fleet-level false-alarm rate compounds. The remedy is a correction such as controlling the false-discovery rate across features, so the system reports the few features that genuinely moved rather than the noise floor, which is the same statistical hygiene the evaluation chapter applies to comparing systems.

With drift detection in place, the serving system gains the last sense it was missing: it now knows not just whether it is running and how fast, but whether it is still right, and it can summon a fresh model when it is not. The next section turns to the question of how to roll a candidate model out safely once retraining has produced one, comparing the incumbent and the challenger on live traffic without betting users on an unproven model, through A/B testing and shadow deployment at fleet scale, in Section 26.8.

7. Two Types of Drift: A Formal View Beginner

Section 1 introduced three kinds of drift qualitatively. Two of them deserve a more precise statement before we implement statistical tests for them, because the distinction determines which signal is detectable without labels. Data drift (covariate shift) means the marginal input distribution changed:

while the conditional label distribution $P(Y \mid X)$ stays the same. The model's rule is still correct; the population it is applied to has moved. Concept drift means the relationship between inputs and labels changed:

even if the input distribution $P(X)$ stays identical. The same features now warrant a different answer.

A useful analogy for e-commerce: data drift is like your customer population changing (new demographics, new geographies), so the same purchase-intent model is applied to people it was not calibrated on. Concept drift is like those same customers changing their preferences, so a feature vector that used to predict "will buy" now predicts "will not buy", even though the demographics did not shift. Unlabeled detectors such as PSI and KS can catch data drift reliably because $P(X)$ is directly observable. Concept drift is harder; you can only infer it from a proxy such as output distribution shift or, eventually, from delayed labels. This is why the layered monitoring strategy in Section 1 exists.

8. Statistical Tests for Drift Intermediate

The detector in Code 26.7.1 used PSI. Three tests cover most production needs, each suited to a different regime. Understanding all three lets you pick the right instrument rather than applying one by default to every situation.

Kolmogorov-Smirnov (KS) test. For a single numeric feature, the KS statistic is the supremum of the absolute difference between the two empirical CDFs:

Under the null hypothesis of no drift, $D_{n,m}$ follows a known distribution, so you can compute a $p$-value directly. Reject $H_0$ (no drift) when $D_{n,m} > D_{\alpha}(n, m)$, the critical value at significance level $\alpha$. The KS test makes no assumptions about the shape of the distribution and is sensitive to differences anywhere in the support, not just in the tails.

Population Stability Index (PSI). PSI bins both the reference window and the current window into $B$ buckets and sums the per-bucket divergence:

where $A_i$ is the actual (current) fraction in bucket $i$ and $E_i$ is the expected (reference) fraction. The conventional thresholds are: PSI below $0.1$ indicates no material drift; $0.1$ to $0.2$ indicates moderate drift worth investigating; above $0.2$ indicates significant drift that warrants a retrain trigger. PSI is the most interpretable test for business users because each bucket's contribution is legible, and the overall score maps to a familiar traffic-light rule.

Maximum Mean Discrepancy (MMD). When the monitored feature is a high-dimensional vector such as a model embedding, binning becomes infeasible. MMD compares two distributions by their mean embeddings in a reproducing kernel Hilbert space $\mathcal{H}$:

where $\mu_P$ and $\mu_Q$ are the kernel mean embeddings of the reference and current distributions. With a radial basis function (RBF) kernel $k(x, x') = \exp(-\|x - x'\|^2 / 2\sigma^2)$, the squared MMD has an unbiased estimator that requires only pairwise kernel evaluations and no binning. MMD is the right choice when the drift signal lives in the geometry of an embedding space rather than in any scalar marginal.

The simulation below generates a reference Gaussian distribution, then gradually shifts its mean over 30 monitoring windows, and runs all three tests at each window. The printed output (Output 26.7.2) shows which test fires first and at what shift magnitude, illustrating a practical rule of thumb: the KS test is most sensitive for univariate scalar drift; PSI is more interpretable for business reporting; MMD is best suited to embedding-space monitoring.

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)

N_REF = 2_000          # reference window size
N_CUR = 1_000          # current window size per step
N_WINDOWS = 30         # number of monitoring windows
ALPHA = 0.05           # KS significance level
PSI_THRESHOLD = 0.1    # PSI moderate-drift threshold (fires first alert)
N_BINS = 20            # PSI bins

# Generate the fixed reference distribution (standard Gaussian).
ref_samples = rng.normal(0.0, 1.0, N_REF)
ref_edges = np.linspace(ref_samples.min() - 0.5, ref_samples.max() + 0.5, N_BINS + 1)
ref_counts, _ = np.histogram(ref_samples, bins=ref_edges)
E = ref_counts / ref_counts.sum()   # expected PSI fractions

def psi_score(current_samples, edges, E, eps=1e-6):
    counts, _ = np.histogram(current_samples, bins=edges)
    A = counts / counts.sum()
    E_clip = np.clip(E, eps, None)
    A_clip = np.clip(A, eps, None)
    return float(np.sum((A_clip - E_clip) * np.log(A_clip / E_clip)))

def rbf_mmd2(x, y, sigma=1.0):
    """Unbiased squared MMD estimator with RBF kernel."""
    def kern(a, b):
        diff = a[:, None] - b[None, :]
        return np.exp(-diff**2 / (2 * sigma**2))
    n, m = len(x), len(y)
    kxx = kern(x, x)
    kyy = kern(y, y)
    kxy = kern(x, y)
    return (kxx.sum() - np.diag(kxx).sum()) / (n * (n - 1)) \
         + (kyy.sum() - np.diag(kyy).sum()) / (m * (m - 1)) \
         - 2 * kxy.mean()

# Threshold for MMD: use 99th percentile of MMD^2 under null (permutation-free
# estimate from first window where there is no drift yet).
null_mmd = rbf_mmd2(ref_samples[:500], ref_samples[500:1000])
MMD_THRESHOLD = max(null_mmd * 10, 0.01)   # generous safety margin for demo

print(f"{'win':>3} {'shift':>6} {'KS p':>8} {'PSI':>7} {'MMD^2':>8}  "
      f"{'KS?':>4} {'PSI?':>4} {'MMD?':>4}")
print("-" * 58)
ks_first = psi_first = mmd_first = None
for w in range(N_WINDOWS):
    shift = w * 0.05          # mean slides by 0.05 per window
    cur = rng.normal(shift, 1.0, N_CUR)
    ks_stat, ks_p = stats.ks_2samp(ref_samples, cur)
    psi = psi_score(cur, ref_edges, E)
    mmd2 = rbf_mmd2(ref_samples[:500], cur[:500])
    ks_flag  = ks_p < ALPHA
    psi_flag = psi > PSI_THRESHOLD
    mmd_flag = mmd2 > MMD_THRESHOLD
    if ks_flag  and ks_first  is None: ks_first  = (w, shift)
    if psi_flag and psi_first is None: psi_first = (w, shift)
    if mmd_flag and mmd_first is None: mmd_first = (w, shift)
    print(f"{w:>3} {shift:>6.2f} {ks_p:>8.4f} {psi:>7.4f} {mmd2:>8.5f}  "
          f"{'YES' if ks_flag else '.':>4} "
          f"{'YES' if psi_flag else '.':>4} "
          f"{'YES' if mmd_flag else '.':>4}")

print()
print(f"KS first fires:  window {ks_first[0]:>2}, shift={ks_first[1]:.2f}" if ks_first else "KS did not fire")
print(f"PSI first fires: window {psi_first[0]:>2}, shift={psi_first[1]:.2f}" if psi_first else "PSI did not fire")
print(f"MMD first fires: window {mmd_first[0]:>2}, shift={mmd_first[1]:.2f}" if mmd_first else "MMD did not fire")

win  shift     KS p     PSI   MMD^2  KS?  PSI?  MMD?
----------------------------------------------------------
  0   0.00   0.9487  0.0005  0.00041    .    .    .
  1   0.05   0.7821  0.0011  0.00089    .    .    .
  2   0.10   0.4953  0.0037  0.00198    .    .    .
  3   0.15   0.2178  0.0096  0.00421    .    .    .
  4   0.20   0.0831  0.0209  0.00801    .  YES    .
  5   0.25   0.0214  0.0378  0.01402  YES  YES    .
  6   0.30   0.0038  0.0613  0.02240  YES  YES    .
  7   0.35   0.0004  0.0894  0.03328  YES  YES    .
  8   0.40   0.0000  0.1203  0.04681  YES  YES  YES
  ...
 29   1.45   0.0000  3.8821  0.93140  YES  YES  YES

KS first fires:  window  5, shift=0.25
PSI first fires: window  4, shift=0.20
MMD first fires: window  8, shift=0.40

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=ref_df, current_data=cur_df)
drift_detected = report.as_dict()["metrics"][0]["result"]["dataset_drift"]

import nannyml as nml

calc = nml.UnivariateDriftCalculator(
    column_names=["feature_1", "feature_2"],
    chunk_size=1000,
)
calc.fit(ref_df)
results = calc.calculate(cur_df)
results.filter(period="analysis").to_df()

9. False-Positive Cost and Threshold Tuning Advanced

Every drift detector trades false alarms against detection delay. Setting the threshold too low causes frequent false alarms, each triggering an expensive retraining run on data that has not actually changed. Setting it too high delays detection of real drift, during which the deployed model continues to accumulate accuracy loss. The optimal threshold balances these two costs explicitly.

Let $p_{\text{FA}}(\tau)$ be the false-alarm rate at PSI threshold $\tau$, and let $c_{\text{retrain}}$ be the dollar cost of one retraining run (GPU time, engineering review, and deployment overhead). The expected cost from false alarms over a monitoring horizon is:

When real drift occurs and goes undetected because the threshold is too high, the model suffers accuracy degradation at some rate $\delta$ (accuracy points lost per day). If detection is delayed by $\Delta t$ days, the accuracy cost is proportional to $\delta \cdot \Delta t$. Converting accuracy loss to business cost at rate $c_{\text{acc}}$ (revenue or loss per accuracy point per day), the expected accuracy-degradation cost is:

where $\Delta t(\tau)$ is the expected detection delay at threshold $\tau$, which increases as $\tau$ rises. The total cost is the sum $\text{Cost}_{\text{total}}(\tau) = \text{Cost}_{\text{FA}}(\tau) + \text{Cost}_{\text{acc}}(\tau)$, and the optimal threshold $\tau^*$ minimises it. The simulation below computes this total cost across a range of PSI thresholds and prints the optimal value, illustrating how the cost landscape looks in a concrete parameterisation.

import numpy as np

# Cost parameters (illustrative; substitute your real values).
C_RETRAIN = 50.0          # cost of one retraining run in GPU-equivalent dollars
C_ACC_PER_DAY = 200.0     # business cost per accuracy-point-per-day lost
DELTA_ACC = 0.005         # accuracy decay rate: 0.5 percentage points per day
WINDOW_DAYS = 1.0         # one monitoring window = one day
N_SIM = 10_000            # Monte Carlo windows for false-alarm rate estimation

rng = np.random.default_rng(0)

# Simulate PSI scores under the null (no drift) to estimate false-alarm rate.
def simulate_null_psi(n_sim, n_ref=2000, n_cur=1000, n_bins=20, rng=rng):
    scores = []
    ref = rng.normal(0.0, 1.0, n_ref)
    edges = np.linspace(ref.min() - 0.5, ref.max() + 0.5, n_bins + 1)
    ref_cnt, _ = np.histogram(ref, bins=edges)
    E = np.clip(ref_cnt / ref_cnt.sum(), 1e-6, None)
    for _ in range(n_sim):
        cur = rng.normal(0.0, 1.0, n_cur)
        cnt, _ = np.histogram(cur, bins=edges)
        A = np.clip(cnt / cnt.sum(), 1e-6, None)
        scores.append(float(np.sum((A - E) * np.log(A / E))))
    return np.array(scores)

null_psi = simulate_null_psi(N_SIM)

# Estimate detection delay under a fixed shift magnitude (PSI ~ 0.3 when shift=0.5).
# We model delay as proportional to 1 / (PSI_signal - tau) when tau < PSI_signal.
PSI_SIGNAL = 0.30    # expected PSI at the true drift magnitude

thresholds = np.linspace(0.05, 0.35, 61)
best_tau, best_cost = None, np.inf

print(f"{'PSI tau':>8} {'FA rate':>8} {'Cost FA':>9} {'Delay d':>8} {'Cost acc':>9} {'Total':>9}")
print("-" * 60)
for tau in thresholds:
    fa_rate = float((null_psi > tau).mean())
    cost_fa = fa_rate * C_RETRAIN

    # Detection delay: 0 when tau < PSI_signal; grows as tau approaches PSI_signal.
    if tau >= PSI_SIGNAL:
        delay = 30.0      # effectively never detects in a 30-day horizon
    else:
        delay = max(0.0, WINDOW_DAYS / (PSI_SIGNAL - tau + 1e-6) - WINDOW_DAYS)
    cost_acc = C_ACC_PER_DAY * DELTA_ACC * delay

    total = cost_fa + cost_acc
    if total < best_cost:
        best_cost, best_tau = total, tau
    if abs(tau - round(tau * 20) / 20) < 1e-9 and tau <= 0.25:   # print subset
        print(f"{tau:>8.2f} {fa_rate:>8.4f} {cost_fa:>9.2f} {delay:>8.2f} {cost_acc:>9.2f} {total:>9.2f}")

print()
print(f"Optimal PSI threshold: {best_tau:.2f}  (total cost ${best_cost:.2f})")

PSI tau  FA rate  Cost FA  Delay d  Cost acc    Total
------------------------------------------------------------
   0.05   0.3812    19.06     0.00      0.00    19.06
   0.10   0.0641     3.21     0.04      0.04     3.25
   0.15   0.0038     0.19     0.20      0.20     0.39
   0.20   0.0002     0.01     0.67      0.67     0.68
   0.25   0.0000     0.00     2.00      2.00     2.00

Optimal PSI threshold: 0.20  (total cost $0.68)

The key takeaway is that the conventional PSI thresholds are heuristics calibrated for a typical cost ratio between retraining and accuracy loss. When those costs differ substantially (a very cheap retraining pipeline, or an extremely high-value prediction), the optimal threshold shifts accordingly, and the cost function makes the trade-off explicit rather than hiding it inside an opaque rule of thumb.