Batch, Streaming, Online, and Interactive AI

"They asked me for yesterday's answer cheaply or today's answer instantly, and seemed genuinely surprised that I could not deliver both at once."

A Pipeline Negotiating Its Latency Budget

In the previous sections we established the thesis (distribute the essential work), the proof that one central form of it is exact (the gradient identity of Section 1.1), and the six axes that organize the book. Those sections asked where the work is split across machines. This section asks a different and equally structural question: when does the work run relative to the data arriving and the answer being needed? A recommendation model trained once a night, refreshed on a streaming feature pipeline every few seconds, and queried interactively at serving time is the same statistical object wearing three different deployment costumes. Choosing the costume well is the difference between a system that is fast, fresh, and affordable and one that is two of those at the expense of the third.

1. The Four Modes, Ordered by Latency Beginner

Batch processing runs the work in large jobs over an accumulated body of data: read a day's worth of logs, retrain or rescore everything, write the results, and stop. Because the per-job overhead (launching workers, reading from storage, scheduling) is amortized over millions of records, batch achieves the highest throughput per dollar of any mode, and the distributed-data machinery of Chapter 6 exists precisely to make these jobs scale across a cluster. The price is freshness: a batch answer reflects the world as it was when the job started, which may be hours or a full day ago.

Streaming processing never stops. Events flow in continuously through a log such as Apache Kafka, and a processor such as Apache Flink or Spark Structured Streaming applies the same computation to each small window of events seconds after they arrive. Streaming trades a little of batch's per-record efficiency (it cannot amortize over a whole day at once) for freshness measured in seconds rather than hours, which is exactly what a fraud feature or a trending-topic counter needs. This is the home of the streaming feature store, and the book develops it in Chapter 9.

Online learning goes one step further: the model parameters themselves change on every event. Where batch and streaming hold the model fixed and only the inputs flow, an online system folds each new labeled example straight into the weights, so the model at time $t+1$ already reflects the example seen at time $t$. This keeps the model continuously adapted to a drifting distribution, at the cost of new failure modes (a single bad batch can corrupt the live model) that Chapter 9 treats alongside streaming, since the two almost always run on the same pipeline.

Interactive serving is request-response under a tight latency budget. A user, or another service, sends one input and waits for the answer, and the entire deadline (often tens to hundreds of milliseconds, or the time to first token for a language model) is the product. Here throughput per dollar is sacrificed deliberately: you keep capacity warm and idle so that any single request is answered now, not when it is convenient. Interactive serving of large models is the subject of Chapter 23 and, for language models specifically, Chapter 24.

2. Why Smaller Groups Cost More: The Amortization Curve Intermediate

The trade-off in Figure 1.5.1 is not a matter of taste; it falls out of a simple cost model. Suppose processing a group of $b$ events carries a fixed overhead $c_0$ (scheduling, reading from storage, launching the model) plus a marginal cost $c_1$ per event. The cost per event is then

As the group size $b$ grows, the fixed overhead $c_0$ is spread thinner and the per-event cost falls toward the floor $c_1$. Batch pushes $b$ into the millions and sits near that floor. Interactive serving runs at $b = 1$, where the full overhead $c_0$ is paid on every single answer. Streaming and micro-batching choose an intermediate $b$ to buy back most of the amortization while keeping latency bounded, because a larger group also means waiting longer to fill it. The code below measures this curve directly by scoring a million events three ways and reporting throughput and per-event latency for each.

import time, numpy as np

rng = np.random.default_rng(0)
N = 1_000_000                      # total events to score
w = rng.standard_normal(16)        # a tiny linear model
X = rng.standard_normal((N, 16))   # the event features

def score(rows):                   # the per-event work, identical in every mode
    return rows @ w

# BATCH: one big matrix multiply over all N events at once.
t0 = time.perf_counter()
_ = score(X)
batch_s = time.perf_counter() - t0

# MICRO-BATCH (streaming): process in windows of 10k events.
t0 = time.perf_counter()
for i in range(0, N, 10_000):
    _ = score(X[i:i+10_000])
stream_s = time.perf_counter() - t0

# INTERACTIVE: one event at a time, the latency a caller actually waits.
t0 = time.perf_counter()
for i in range(2_000):             # sample 2k single-event calls
    _ = score(X[i:i+1])
inter_s = (time.perf_counter() - t0) / 2_000

print(f"{'mode':<14}{'throughput (events/s)':>24}{'per-event latency (us)':>26}")
print(f"{'batch':<14}{N/batch_s:>24,.0f}{batch_s/N*1e6:>26.2f}")
print(f"{'streaming':<14}{N/stream_s:>24,.0f}{stream_s/N*1e6:>26.2f}")
print(f"{'interactive':<14}{1/inter_s:>24,.0f}{inter_s*1e6:>26.2f}")

mode             throughput (events/s)    per-event latency (us)
batch                       65,915,668                      0.02
streaming                   61,711,634                      0.02
interactive                    783,239                      1.28

The numbers make the amortization curve concrete. Batch and streaming, with large $b$, sit near the marginal-cost floor and reach tens of millions of events per second. The interactive mode, at $b = 1$, pays the fixed overhead on every call and its throughput collapses by nearly two orders of magnitude, even though the arithmetic per event is identical. This is the entire reason interactive serving is expensive: you are not doing harder work per event, you are doing it in the least amortizable way possible, because a waiting caller cannot be told to come back once a million friends have queued up behind them. Real serving systems claw back some of this loss with dynamic batching, grouping concurrent requests on the fly, which is one of the central techniques of Chapter 24.

3. The Modes Compose Inside One System Beginner

It would be a mistake to read Figure 1.5.1 as four boxes you choose between. A production AI system almost always runs several modes at once, each on the part of the workload that fits it. Table 1.5.1 lays out a single recommendation service to make the point: the same underlying model is trained in batch, kept fresh in streaming, optionally nudged online, and queried interactively, and each row sits at a different point on the freshness-cost curve for a good reason.

The composition is the design. The expensive, throughput-hungry full retrain runs once a night in batch because nobody needs last-second weights for the bulk of the model. The features that must reflect a click from ten seconds ago ride a streaming pipeline. A genuinely fast-moving signal, a suddenly viral item, may justify an online update so the model adapts within minutes rather than waiting for the next night. And the final ranking, which a user is actively waiting on, runs interactively under a tight budget. Each mode addresses a different freshness requirement, and forcing the whole system into any single mode would be wasteful at one end and too stale at the other. Recognizing this layered structure, and the distributed machinery each layer needs, is what the rest of the book equips you to do.

4. Latency, Freshness, and the Cost of Waiting Intermediate

It is worth separating two ideas that the axis in Figure 1.5.1 deliberately bundles, because they can diverge. Latency is how long a caller waits for one answer once they ask. Freshness (or its inverse, staleness) is how old the data and model behind that answer are. Interactive serving has low latency but can still serve a stale model if that model was last trained in last night's batch job; the request is answered in milliseconds, yet the weights are a day old. Streaming attacks freshness without necessarily attacking serving latency: it keeps the features current but the request itself may still be a separate interactive call. Keeping the two notions distinct lets you diagnose a complaint precisely. "The answer was slow" is a latency problem; "the answer was instant but wrong because it ignored what just happened" is a freshness problem, and they have entirely different remedies.

There is also a hard floor on freshness set by physics and queueing, not by laziness. To process events in groups of $b$, a streaming system must wait for $b$ events to arrive, so at an arrival rate of $\lambda$ events per second the window alone adds on the order of $b / \lambda$ seconds of delay before any processing begins. Shrinking $b$ toward one removes that wait but, as Output 1.5.1 showed, surrenders the amortization that kept the work cheap. This is the freshness-versus-throughput trade-off restated in time rather than dollars, and it is why "just make it real-time" is never free: every second of freshness you demand is paid for in efficiency you give up. Section 1.6 turns latency, throughput, cost, and reliability into the four quantities you will measure for the rest of the book.

5. Where Each Mode Lives in the Book Beginner

The four modes are not just a conceptual frame; each anchors a substantial part of the book, and naming the destinations now lets later chapters assume this vocabulary. Batch processing at cluster scale is built from the MapReduce model up in Chapter 6, the foundation that Spark then accelerates. Streaming and online learning share a home in Chapter 9, because in practice a streaming dataflow and a per-event model update run on the same pipeline and face the same questions of windowing, state, and fault tolerance. Interactive serving splits by model size: general distributed inference systems are Chapter 23, and the specialized world of serving large language models, with its token-by-token latency and dynamic batching, is Chapter 24. The library shortcut below shows just how little code stands between you and a real streaming consumer, the entry point to that whole arc.

from kafka import KafkaConsumer   # pip install kafka-python
import json

consumer = KafkaConsumer(
    "transactions",                              # the topic of incoming events
    bootstrap_servers="broker:9092",
    group_id="fraud-scorer",                     # the library manages partition assignment
    value_deserializer=lambda b: json.loads(b),
)

for msg in consumer:                             # blocks until the next event arrives
    event = msg.value
    risk = score(features_for(event))           # the same model, now scoring a live stream
    emit(event["id"], risk)                      # seconds after the event was produced

With the processing modes named and placed, we have answered when the work runs. What remains is to make the trade-offs measurable. Every claim in this section, "cheaper per record," "lower latency," "fresher," became a quantity we waved at rather than measured. The next section fixes that by defining throughput, latency, cost, and reliability precisely, so that the freshness-versus-throughput curve of Figure 1.5.1 becomes something you can put numbers on for any system you build. That treatment begins in Section 1.6.