Large-Model Loading, Cold Starts, and Warm Pools

"I was scheduled in eight hundred milliseconds. Then I spent four minutes reading my own weights off a disk, and the traffic spike I was summoned to handle had already given up and gone home."

A Replica That Booted Too Late

The earlier sections of this chapter treated a replica as a unit you can add or remove: Section 23.5 packed several models onto shared GPUs, and the autoscaler of Section 23.4 grew and shrank the replica count in response to utilization and queue depth. Both implicitly assumed that adding a replica is cheap and fast, the way adding a stateless web server behind a load balancer is cheap and fast. For large models that assumption is false. A web server ships a few megabytes of code and serves its first request in well under a second. A modern inference replica must move the model itself into GPU memory before it can answer anything, and the model is the largest object in the system. This single fact, that the serving unit carries gigabytes of state that must be materialized on the device, reshapes how a serving fleet scales, fails over, and bills.

The size of the gap is easy to underestimate. A 7-billion-parameter model in half precision is roughly fourteen gigabytes of weights; a 70-billion-parameter model is around a hundred and forty gigabytes that must be sharded across several GPUs before any of them can serve; the largest deployed models reach several hundred gigabytes. Pulling that many bytes from object storage, placing them on the right devices, building the CUDA context, and warming the kernels is not a rounding error against a sub-second scheduling decision. It is the dominant term, and a fleet that ignores it will autoscale into a spike it has already missed.

1. Where the Cold-Start Time Goes Beginner

It helps to break the cold start into its contributing costs, because each one is owned by a different part of the system and has a different remedy. The first cost is pulling the weights. The model lives in object storage or a registry, and the new node must read every byte across the network before it can place anything on a GPU. At a per-node read bandwidth of $B$ bytes per second for a model of $S$ bytes, this phase alone takes $S / B$ seconds, and for a multi-hundred-gigabyte model over a shared network link that is minutes, not seconds. This is the same storage-and-loading bottleneck that Section 8.2 analyzed for training data; here the object being streamed is the model rather than the dataset, but the bandwidth arithmetic is identical.

The second cost is loading and sharding the weights onto the devices. Bytes that have arrived on the host still have to be copied over the PCIe or NVLink bus into GPU memory, and for a model too large for one GPU they must be split across several according to a tensor-parallel or pipeline plan before any device holds a runnable slice. That partition is exactly the sharded-loading problem of Chapter 16: the same layout that lets a 70-billion-parameter model fit across eight GPUs at serving time also dictates how its checkpoint is read and scattered at load time. The third cost is building the CUDA context and warming the kernels: the first forward pass triggers just-in-time compilation, autotuning, and CUDA-graph capture, and any prefix or KV cache the engine relies on starts empty and cold. That warmup is the per-node startup cost detailed in Section 22.8, and until it completes the replica's first few requests run slow even after the weights are in place.

Putting the phases together, the cold-start latency is

and the binding term shifts with the model. For a small model on fast local storage, warmup dominates; for a hundred-gigabyte model behind a shared object store, the pull term $S/B$ swamps everything else. Knowing which term binds tells you which mitigation to reach for, exactly as Chapter 1 insisted you match the remedy to the ceiling that actually binds.

2. Why Cold Starts Break Autoscaling and Serverless GPU Intermediate

An autoscaler observes a signal (utilization, queue depth), decides it needs more replicas, and requests them. With stateless services the requested capacity is usable within a second, so the control loop is tight and the lag between deciding and serving is negligible. With large-model replicas the requested capacity is usable only after $T_{\text{load}}$, which inserts a fixed dead time into the control loop. During a spike that dead time is precisely when you most need the capacity, and the replicas you summoned arrive after the leading edge of the spike has already overwhelmed the replicas you had. This is the mechanism behind the autoscaling lag flagged in Section 23.4: the loop is not slow because the autoscaler is timid; it is slow because the thing it creates takes minutes to exist.

The same dead time is what makes serverless GPU inference hard. The serverless promise is scale to zero: hold no replicas when idle, spin one up on the first request, and pay nothing in between. For a stateless function that is a clean win because spin-up is milliseconds. For a large model, scale to zero means every first request after an idle period pays the full $T_{\text{load}}$, so the user who triggers the wake waits seconds to minutes for an answer. Scale to zero is therefore viable only for workloads that tolerate that first-request latency: batch jobs, internal tools, low-traffic endpoints where a cold first request is acceptable. For latency-sensitive serving the honest choice is to keep capacity warm, which means the next section's idea.

3. Warm Pools and the Cost-Versus-Readiness Trade-Off Intermediate

The direct remedy for a fixed cold-start dead time is to pay it in advance. A warm pool keeps $K$ spare replicas fully loaded, warmed, and idle above the baseline you need for steady traffic, so when a spike arrives the capacity is already on the device and serves the very first request. The pool converts an unavoidable startup latency into a standing cost: those $K$ replicas hold GPUs and bill for them whether or not a spike ever comes. That is the cost-versus-readiness trade-off in one sentence, and it is a genuine business decision rather than a purely technical one. A pool sized for the worst spike meets the latency objective always but wastes money during the long stretches of normal traffic; a pool sized too small saves money but lets some spikes through to the cold-start dead time.

You can put numbers on it. If a spike of size $\Delta$ requests per second arrives and one ready replica serves $c$ requests per second, you need $\lceil \Delta / c \rceil$ spare replicas warm to absorb it without ever touching the cold path. The idle cost is those replicas times their hourly GPU price times the fraction of time the spike is absent. Sizing the pool is then a quantitative balance between the dollars of idle GPUs and the dollars (or SLO credits) of violated latency, the same kind of cost model Chapter 3 built for communication. The demo below makes the readiness side of that balance vivid by simulating a spike against both strategies.

The simulation runs a one-second-tick queue for two minutes. Steady traffic of 40 requests per second jumps to 200 for a fifty-second spike, each ready replica serves 30 requests per second, and a cold replica needs 25 seconds to become ready. The SLO requires every request to begin service within two seconds. We compare pure cold-start autoscaling, which starts with no spares and requests cold replicas once the queue builds, against a warm pool that holds six spare loaded replicas above baseline.

import random
from collections import deque

random.seed(7)
HORIZON, BASE_RPS, SPIKE_RPS = 120, 40, 200      # seconds; arrivals/sec before, during
SPIKE_START, SPIKE_END = 20, 70                  # spike window (seconds)
CAP_PER_REP, T_LOAD, W_SLO, N_BASE = 30, 25, 2, 2  # serve rate, cold-start sec, SLO sec, always-on

def arrivals(t):                                 # rough Poisson arrivals this tick
    rate = SPIKE_RPS if SPIKE_START <= t < SPIKE_END else BASE_RPS
    return sum(1 for _ in range(rate * 3) if random.random() < 1.0 / 3.0)

def simulate(warm_spares, autoscale):
    ready = N_BASE + warm_spares                 # replicas able to serve right now
    pending, requested, q = [], 0, deque()       # loading replicas, count requested, arrival ticks
    total = late = 0
    for t in range(HORIZON):
        ready += sum(1 for r in pending if r == t)        # cold replicas finish loading
        pending = [r for r in pending if r > t]
        if autoscale and len(q) > CAP_PER_REP:            # autoscaler reacts to queue depth
            want = len(q) // CAP_PER_REP
            while requested < want:                        # but each new replica lags by T_LOAD
                pending.append(t + T_LOAD); requested += 1
        a = arrivals(t); total += a
        for _ in range(a): q.append(t)                    # enqueue arrivals with their time
        for _ in range(min(ready * CAP_PER_REP, len(q))): # serve up to capacity
            if t - q.popleft() > W_SLO: late += 1          # started service too late?
    late += sum(1 for arr in q if HORIZON - arr > W_SLO)   # drain leftover queue
    return total, late, ready

print(f"{'strategy':34s} {'requests':>8s} {'SLO_viol':>9s} {'viol_%':>7s} {'replicas_end':>13s}")
for label, spares in [("cold-start autoscaling (0 spare)", 0), ("warm pool (6 spare replicas)", 6)]:
    tot, late, endready = simulate(spares, autoscale=True)
    print(f"{label:34s} {tot:8d} {late:9d} {100.0*late/tot:6.1f}% {endready:13d}")

strategy                           requests  SLO_viol  viol_%  replicas_end
cold-start autoscaling (0 spare)      12804      6022   47.0%           124
warm pool (6 spare replicas)          12875         0    0.0%             8

The contrast is stark and it is the whole point of the section. The cold-start strategy is not lazy; its autoscaler fires hard, eventually requesting so many replicas that it ends with 124 of them, far more than the warm pool's eight. But because every one of those replicas needed 25 seconds to load, they all arrived after the spike's leading edge had already blown the SLO, and the late flood then over-provisions wildly. The warm pool spent six idle replicas' worth of GPU money to hold capacity ready, and in return served the entire spike without a single violation. That is the trade purchased: standing idle cost in exchange for capacity that exists at the instant of demand rather than $T_{\text{load}}$ seconds after it.

4. Shrinking the Cold Start Itself Advanced

Warm pools pay the cold start in advance; the complementary family of techniques makes the cold start itself smaller, so that when you do load a replica it becomes ready in less time. The most direct lever is the pull term $S/B$. Caching weights on local NVMe means the second and later replicas on a node read from a fast local disk rather than re-pulling from object storage across the network, turning minutes into seconds for warm-cache nodes. Fast weight streaming overlaps the pull with the load: rather than downloading the whole checkpoint and then copying it to the GPU, the engine streams shards and begins placing the earliest layers on the device while later layers are still arriving, so the two slowest phases run concurrently instead of in series.

A second lever attacks warmup. Snapshotting a warmed process captures a replica after it has loaded and warmed, freezing the hot CUDA context and caches, so a new replica restores from the snapshot rather than recompiling kernels and recapturing graphs from scratch. Lazy and parallel loading help on both fronts: lazy loading defers weights the first request does not touch, letting the replica serve before it is fully materialized, and parallel loading reads many shards across many threads or many GPUs at once so the load phase is bounded by aggregate bandwidth rather than a single stream. None of these eliminates $T_{\text{load}}$, but each shrinks a term in it, which both lowers the first-request latency for scale-to-zero workloads and lets a warm pool be smaller for the same readiness, because a faster cold path is a cheaper backstop when the pool is exhausted.

# Ray Serve: keep a warm floor of replicas always loaded, autoscale above it.
from ray import serve

@serve.deployment(
    autoscaling_config={
        "min_replicas": 8,          # the warm pool: always loaded and ready
        "max_replicas": 40,         # ceiling the autoscaler may grow to
        "target_ongoing_requests": 12,
        "downscale_delay_s": 600,   # grace period so we do not pay T_load to re-add
    },
    ray_actor_options={"num_gpus": 1},
)
class LLM:
    def __init__(self):
        self.engine = load_and_warm_model()   # the slow T_load, paid once at replica start
    async def __call__(self, request):
        return await self.engine.generate(request)

5. Choosing a Strategy for a Given Workload Intermediate

The strategies are not rivals; they are points on a spectrum from cheapest-idle to fastest-ready, and the right point depends on the workload's tolerance for a cold first request. A latency-critical interactive endpoint with a strict SLO and spiky traffic sits at the fast-ready end: a warm pool sized to the expected spike, backed by fast streaming and local caches so even a pool miss recovers quickly. A throughput-oriented batch endpoint, where requests queue anyway and no human waits on the first token, sits at the cheap-idle end: scale to zero is correct, and paying $T_{\text{load}}$ once at the start of a batch is negligible against the hours the batch runs. Most production services live in between, with a small warm floor for the common case and autoscaling above it for genuine surges.

The decision also threads into the next section. A warm pool is not only a performance device; it is a redundancy device, because spare loaded replicas are exactly what lets a fleet survive the loss of a serving replica without a cold-start gap while a replacement loads. Section 23.7 takes up that connection directly, treating availability, failover, and the redundant capacity that keeps a serving fleet answering through failures. The cold-start cost you learned to price here is the same cost that failover must hide, and the warm pool you sized for spikes is the same pool that absorbs a failed replica.

We have located where cold-start time goes, shown why that fixed dead time lags autoscaling and frustrates serverless GPU, priced the warm pool's trade of idle cost for readiness with a simulation, and surveyed the techniques that shrink the cold start itself. The capacity you now know how to keep warm and ready is also the capacity that keeps a fleet available when a replica fails, which is where the chapter turns next.