Distributed Real-Time Inference Pipelines

"They asked me to score every event the instant it arrived. Then they asked me to do it twice as fast. I asked them, politely, to send the events twice as slowly."

A Model Operator Holding the Line on Latency

The previous sections gave us the ingredients. Section 9.5 put events on a partitioned, replayable log so they survive a consumer crash. Section 9.7 showed how to compute features online so a model sees fresh signal rather than yesterday's batch aggregate. What remains is to place a model in the middle of that flow and let its predictions drive action while the events are still warm. A fraud score is worthless an hour after the transaction cleared; a recommendation is worthless after the user has left the page. Real-time inference is the discipline of scoring events fast enough that the decision still matters, across machines that fail independently and a load that never politely pauses.

1. Embed the Model or Call a Service Beginner

The first design choice is where the model actually runs relative to the stream. In the embedded pattern, the model weights are loaded inside the stream-processing operator itself, so scoring is an ordinary in-process function call on the same machine that already holds the event. In the service pattern, the operator makes a network call to a separate model-serving deployment (a microservice fronted by a load balancer), which holds the weights and may scale independently. The two patterns trade the same currency that the whole book trades: latency against coupling.

Embedding is fast because there is no network hop and no serialization; the prediction is a function call away, and the per-event latency is just the model's compute time. The cost is coupling. The model now ships, scales, and fails together with the stream job; updating the model means redeploying the pipeline, and a large model multiplies the memory of every operator instance. Calling a service decouples the two: the serving fleet scales to its own load, the model is updated without touching the pipeline, and a GPU-backed model is shared across many stream consumers instead of replicated into each. The cost is a network round trip on the critical path and a second system that can fail or slow independently. That serving fleet, with its batching, replication, and autoscaling, is a subject in its own right; we build it in Chapter 23 and specialize it for large language models in Chapter 24. Here we only need the trade-off, because it decides the shape of the pipeline.

2. Backpressure: When the Model Is Slower Than the Stream Intermediate

A stream does not wait. Events arrive at whatever rate the world produces them, and that rate is not under your control. The model, by contrast, has a fixed throughput: it can score only so many events per second. When the arrival rate exceeds the service rate, the gap has to go somewhere, and there are only three places it can go. It can pile up in a buffer (which grows without bound until memory runs out), it can be discarded (which silently loses events and corrupts every downstream count), or it can push back on the source so the source slows down. The third option is backpressure, and it is the only one that is both bounded and lossless.

The arithmetic is unforgiving. If events arrive at rate $\lambda$ and the model serves them at rate $\mu$, then whenever $\lambda > \mu$ the queue depth grows on average at $\lambda - \mu$ per unit time. A buffer of capacity $C$ overflows after roughly $C / (\lambda - \mu)$ seconds of sustained overload, no matter how large $C$ is. You cannot buffer your way out of a rate mismatch; you can only delay the reckoning. Backpressure converts that doomed race into a stable control loop: as the queue fills toward a high-water mark, the consumer signals the source to slow its emission, and as the queue drains below a low-water mark, the source is released back to full speed. The queue oscillates inside its bounds and nothing is lost.

The simulation below builds exactly this loop in pure Python: a producer whose peak rate is twice the model's service rate, a bounded queue, and a hysteresis controller that throttles the producer between a high- and low-water mark. We watch the queue grow, the throttle engage, and the drop count stay at zero. The model in the loop is a stand-in (any scoring function would do); the point is the control behavior of the pipeline around it.

import collections
import random

random.seed(7)

# --- fixed system parameters (one "tick" = 1 millisecond of virtual time) ---
TICKS = 4000                 # total virtual milliseconds simulated
QUEUE_CAP = 200              # hard buffer ceiling (events)
HIGH_WATER = 150             # throttle the producer at/above this depth
LOW_WATER = 50               # release the throttle at/below this depth
MODEL_MS = 4.0               # the model scores one event every 4 ms => 250 ev/s

# Peak demand (0.50/ms = 500 ev/s) is TWICE the model's 250 ev/s service rate:
# the pipeline is structurally overloaded and MUST apply backpressure.
PEAK_RATE = 0.50
THROTTLED_RATE = 0.15        # rate the producer falls back to under backpressure

queue = collections.deque()
throttled = False
next_model_free = 0.0        # virtual time when the model can take the next event
produced = scored = dropped = 0
peak_depth = backpressure_ticks = 0
samples = []

for tick in range(TICKS):
    depth = len(queue)
    if depth >= HIGH_WATER:          # hysteresis controller: two thresholds
        throttled = True
    elif depth <= LOW_WATER:
        throttled = False
    if throttled:
        backpressure_ticks += 1

    rate = THROTTLED_RATE if throttled else PEAK_RATE   # source obeys the signal
    if random.random() < rate:
        if len(queue) < QUEUE_CAP:
            queue.append(tick); produced += 1
        else:
            dropped += 1; produced += 1                 # last-resort drop only

    if tick >= next_model_free and queue:               # model scores one event
        queue.popleft(); scored += 1
        next_model_free = tick + MODEL_MS

    peak_depth = max(peak_depth, len(queue))
    if tick % 250 == 0:
        samples.append((tick, len(queue), throttled))

print("model service rate     :", f"{1000.0 / MODEL_MS:.0f} events/s")
print("producer peak rate     :", f"{PEAK_RATE * 1000:.0f} events/s  (2x the model)")
print()
print("tick    queue_depth   producer_state")
for t, d, thr in samples:
    print(f"{t:>5}   {d:>5} {'#' * (d // 5):<32} {'THROTTLED' if thr else 'full-rate'}")
print()
print("events produced        :", produced)
print("events scored          :", scored)
print("events dropped (buffer) :", dropped)
print("peak queue depth       :", f"{peak_depth}  (cap {QUEUE_CAP})")
print("ticks under backpressure:",
      f"{backpressure_ticks} / {TICKS}  ({100.0 * backpressure_ticks / TICKS:.0f}% of the run)")
print("drop rate              :", f"{100.0 * dropped / produced:.2f}% of produced events")

model service rate     : 250 events/s
producer peak rate     : 500 events/s  (2x the model)

tick    queue_depth   producer_state
    0       0                                  full-rate
  250      71 ##############                   full-rate
  500     134 ##########################       full-rate
  750     129 #########################        THROTTLED
 1000     106 #####################            THROTTLED
 1250      90 ##################               THROTTLED
 1500      56 ###########                      THROTTLED
 1750      95 ###################              full-rate
 2000     145 #############################    THROTTLED
 2250     110 ######################           THROTTLED
 2500      88 #################                THROTTLED
 2750      66 #############                    THROTTLED
 3000      77 ###############                  full-rate
 3250     141 ############################     full-rate
 3500     127 #########################        THROTTLED
 3750      95 ###################              THROTTLED

events produced        : 1078
events scored          : 1000
events dropped (buffer) : 0
peak queue depth       : 151  (cap 200)
ticks under backpressure: 2676 / 4000  (67% of the run)
drop rate              : 0.00% of produced events

The trace tells the whole story. The queue climbs while the producer runs full-rate, crosses 150, and the controller engages; the model drains the backlog until the depth falls below 50 and the producer is released, then the cycle repeats. The system spends two thirds of its time under backpressure because the offered load genuinely exceeds capacity, yet the drop rate is zero: the source absorbed the mismatch by slowing down, exactly as Figure 9.8.1 promises. A pipeline without this loop would have grown the queue without bound and either crashed on memory or begun discarding events the moment the buffer filled.

# Faust (a Python stream processor in the Kafka ecosystem). The async loop
# only pulls the next event when the model coroutine is ready for it, so an
# overloaded model naturally stops consuming and Kafka stops advancing the
# consumer offset. Backpressure is the absence of a faster pull, not extra code.
@app.agent(transactions_topic)
async def score(stream):
    async for event in stream:          # pulls one event only when ready
        features = await enrich(event)  # online feature join (Section 9.7)
        prediction = model.score(features)
        await actions_topic.send(value=prediction)   # emit the decision

3. Delivery Guarantees: Exactly-Once Versus At-Least-Once Actions Advanced

An inference pipeline does not just compute predictions; it emits actions, and actions have consequences in the world. The delivery guarantee on that emission is therefore a product decision, not a systems detail. Consider the three guarantees a streaming system can offer for the action it fires per event. At-most-once means an action may be skipped but never repeated: tolerable for a best-effort recommendation, catastrophic for a fraud block that lets the fraud through. At-least-once means an action may be repeated but never skipped: every fraud is caught, but a customer may receive the same alert twice or a card may be declined twice on a retry. Exactly-once means each event drives precisely one action, which is what everyone wants and what costs the most to guarantee.

The reason exactly-once is expensive is failure. A pipeline that crashes after scoring an event but before recording that it acted will, on recovery, replay the event from the log (Section 9.5) and is now at risk of acting twice. At-least-once is the natural default of a replayable log plus retries; you get it almost for free, and you pay for it in duplicates. Exactly-once requires that the act of emitting and the act of recording progress commit together atomically, so a crash can never leave one done without the other. Kafka's transactional producer offers this as a read-process-write transaction, and Flink achieves it with a two-phase commit tied to its checkpoints. The cheaper and more common engineering answer is to keep at-least-once delivery and make the action idempotent: tag each action with the event's unique id and have the downstream consumer ignore an id it has already applied, so a duplicate emission collapses into a single effect.

4. Holding the End-to-End Latency Budget Intermediate

A real-time pipeline lives or dies by a service-level objective stated end to end, usually as a tail percentile: "99% of events produce an action within 50 milliseconds of arrival." That budget is spent across every hop in Figure 9.8.1, and the hops add. If $T$ is the time from event arrival to action emission, then

where $t_{\text{queue}}$ is time spent waiting in buffers, $t_{\text{feature}}$ is the online feature join, $t_{\text{model}}$ is scoring, and $t_{\text{emit}}$ is writing the action downstream. The objective is on the tail of $T$, not its mean, and tails compose punishingly: a stage that is fast at the median but occasionally slow (a feature lookup that misses cache, a model that hits a garbage-collection pause) inflates the tail of the sum even when every mean looks healthy. This is why backpressure matters for the budget and not only for stability: an unbounded queue turns $t_{\text{queue}}$ into the dominant, ever-growing term, and the SLO is violated long before memory runs out. The methods for measuring and attributing tail latency across a distributed pipeline are the subject of Chapter 5, and the cost models that predict each term come from Chapter 3.

The practical consequence is that the embed-versus-serve choice from Section 1 is partly a latency-budget choice. An external service adds a network round trip to $t_{\text{model}}$, often a few milliseconds at the median but tens of milliseconds at the tail under load, which can be the difference between meeting and missing a tight budget. Request batching on the serving side, which Chapter 23 develops, trades a little median latency for much higher throughput, and whether that trade fits depends entirely on where your budget binds.

We now have a complete real-time pipeline: a model placed by the embed-or-serve trade-off, a bounded queue with backpressure so a slow model cannot blow the buffer, a delivery guarantee chosen by which error hurts, and a latency budget tracked across every hop. What we have not yet asked is whether the model is still right. A pipeline can be fast, lossless, and exactly-once while quietly scoring against a world that has drifted out from under it. Detecting that drift across a fleet of streaming consumers is where Section 9.9 goes next.