Cost Control Across the Fleet

"A token, costing a fraction of a cent, multiplied by a billion, is suddenly a budget meeting. Nobody asked me to multiply; I just kept generating."

A Token, Unaware of Its Own Arithmetic

An individual agent call is cheap. A single request to a large model, eighteen hundred tokens in and a few hundred out, costs about a penny, and at that price cost feels like a rounding error next to the engineering effort of getting the agent to behave. That intuition survives exactly until the system is deployed across a fleet. The agents of Section 40.6 and Section 40.7 do not make one call; each task drives a loop of planning, tool use, reflection, and retry, and the fleet runs millions of those tasks a day. The penny is multiplied by the loop length, then by the task volume, and the rounding error becomes the largest variable cost in the system. Controlling that number is the subject of this section, and it draws directly on the MLOps cost discipline of Chapter 26, the per-node serving efficiency of Chapter 22, and the spot and build-versus-buy economics of Section 33.8 and Section 33.9.

1. The Cost Model Is Multiplicative Beginner

The reason fleet cost surprises people is that it is a product of factors, not a sum, and products grow faster than intuition expects. Let a fleet handle $R$ tasks per period, let each task run an agent loop of $S$ model calls on average, and let each call consume $t_{\text{in}}$ input tokens and $t_{\text{out}}$ output tokens at per-token prices $p_{\text{in}}$ and $p_{\text{out}}$. The total cost of the period is

Every term is a multiplier. Doubling the task volume doubles the bill; so does doubling the loop length, the context size, or the price of the model. The factors that practitioners underweight are the two in the middle. Agent loops multiply the call count: a task that a human would phrase as one question becomes six model calls once the agent plans, calls a tool, reads the result, reflects, and retries, so the realized $S$ is far above one. Long contexts multiply the input cost: retrieved documents, conversation history, and tool schemas push $t_{\text{in}}$ into the thousands, and because input tokens are billed on every call of the loop, a large context is paid for repeatedly within a single task. The output term is usually smaller in token count but carries a higher unit price, so it cannot be ignored. The combination is what makes a one-cent call into a seven-figure annual line, and it is why cost belongs in the system metrics of Chapter 3 alongside latency and throughput.

2. A Map of the Levers Beginner

Each lever in this section reduces one factor of the cost product, and they are best understood as stations on the path a request travels from arrival to a billed model call. Figure 40.8.1 lays out that path: a request first meets a cache that may answer it outright, then a router that sends it to a small model and escalates to a large one only when needed, then a serving layer whose per-token cost is set by quantization and batching. Each station is annotated with the factor it cuts, so the diagram doubles as a checklist for where a fleet's money actually goes.

3. Routing and Cascades: Pay the Big Model Only When You Must Intermediate

The single largest lever is usually the model price, because the gap between a small and a large model is an order of magnitude or more per token, while the small model handles a large share of calls acceptably. A cascade exploits this by trying the cheap model first and escalating to the expensive one only when a confidence check, a verifier, or the small model's own self-report says the answer is inadequate. If the small model costs $c_S$ per call, the large model costs $c_L$, and a fraction $q$ of calls escalate, the expected cost per call under the cascade is

where the $c_S$ is paid on every call (the cheap attempt always runs) and the $c_L$ is paid only on the escalating fraction. When $c_S \ll c_L$ and $q$ is well below one, this expected cost sits far below the $c_L$ that a large-model-only fleet pays on every call. The design problem is to push $q$ down without letting quality fall, which is exactly the routing and confidence-estimation work of the orchestration layer in Chapter 32. A poorly calibrated router that escalates everything recovers the full large-model bill plus the wasted small-model attempts, so the cascade is only as good as its escalation decision.

import functools, hashlib

cache = {}                                  # in prod: Redis or a semantic vector cache

def cached(fn):                             # whole-answer cache keyed on the prompt
    @functools.wraps(fn)
    def wrap(prompt):
        key = hashlib.sha256(prompt.encode()).hexdigest()
        if key in cache:
            return cache[key]               # cache hit: zero model tokens billed
        out = cache[key] = fn(prompt)
        return out
    return wrap

@cached
def answer(prompt):
    small = call_small(prompt)              # cheap model first
    if confidence(small) >= 0.8:            # router's escalation gate
        return small
    return call_large(prompt)               # escalate only the hard cases

4. Caching, Compression, and Loop Budgets Intermediate

Caching attacks the call count from two directions. Semantic caching stores whole answers and serves a near-identical future request from the store, so a cache hit costs only an embedding lookup rather than a full generation; if a fraction $h$ of calls hit and a hit costs a fraction $\beta$ of a full call, the effective billable call count falls to $R S (1 - h + h\beta)$, and the saving

is close to $h$ whenever the lookup is cheap. Prefix and KV caching attack the same calls from inside: the long shared prefix of an agent prompt, the system instructions, tool schemas, and retrieved context, is encoded once and its key-value state reused across the loop, so the per-call input cost on a provider that bills cached input at a discount drops sharply. This is the fleet-scale return on the KV-cache and prefix-reuse machinery of Chapter 22 and the distributed serving of Chapter 24; a per-node efficiency that looked like a single-machine optimization is multiplied across every call of every agent in the fleet. Context compression complements caching by shrinking $t_{\text{in}}$ itself: summarizing history, pruning low-relevance retrieved chunks, and dropping verbose tool output trims the prompt that is billed on every loop step.

The last lever is defensive rather than optimizing. An agent loop can run away, retrying, re-planning, and re-calling tools without converging, and a single runaway task can cost a thousand times a normal one. A hard step budget that caps $S$ at a maximum, combined with a token budget per task and a wall-clock timeout, bounds the worst case and protects the bill from the long tail. Because the cost model multiplies by $S$, capping $S$ caps the blast radius of any pathological agent directly. These budgets are the cost-side complement to the reliability budgets that the MLOps practice of Chapter 26 already tracks per service.

5. Putting the Levers Together Intermediate

The levers compose, and the only honest way to see how much they save is to model the whole product rather than reason about one factor at a time. Code 40.8.2 builds a fleet workload from the cost model of Section 1, then applies the cache of Section 4 and the cascade of Section 3 in sequence, reporting the cost after each. It is deliberately concrete: real per-million-token prices, a realistic loop length, and a context size in the low thousands.

import math

# ---- Fleet workload parameters (one day) ----
tasks_per_day = 2_000_000        # agent tasks across the fleet in one day
steps_per_task = 6               # mean LLM calls per agentic loop
in_tokens = 1800                 # mean prompt tokens per call (long context + history)
out_tokens = 350                 # mean generated tokens per call
price_in_L, price_out_L = 3.00, 15.00        # large-model API price, USD / 1M tokens

def call_cost(p_in, p_out, n_in, n_out):
    return n_in * p_in / 1e6 + n_out * p_out / 1e6

# Baseline: every call hits the large model.
cost_call_L = call_cost(price_in_L, price_out_L, in_tokens, out_tokens)
calls_per_day = tasks_per_day * steps_per_task
baseline = calls_per_day * cost_call_L

# Lever 1 - semantic cache: a fraction of calls answered from the store.
cache_hit, cache_cost_frac = 0.35, 0.02
eff_calls = calls_per_day * (1 - cache_hit + cache_hit * cache_cost_frac)
after_cache = eff_calls * cost_call_L

# Lever 2 - small-then-large cascade on the uncached calls.
price_in_S, price_out_S = 0.15, 0.60
escalation_rate = 0.25
cost_call_S = call_cost(price_in_S, price_out_S, in_tokens, out_tokens)
cost_call_cascade = cost_call_S + escalation_rate * cost_call_L   # E[c] = c_S + q c_L
uncached = calls_per_day * (1 - cache_hit)
cached_residual = calls_per_day * cache_hit * cache_cost_frac * cost_call_L
after_cascade = uncached * cost_call_cascade + cached_residual

# Lever 3 - context compression on the small model's prompt.
compress = 0.55
cost_call_S_c = call_cost(price_in_S, price_out_S, in_tokens * compress, out_tokens)
after_all = uncached * (cost_call_S_c + escalation_rate * cost_call_L) + cached_residual

for label, c in [("baseline (all large)", baseline), ("+ cache", after_cache),
                 ("+ cache + cascade", after_cascade), ("+ all three levers", after_all)]:
    print(f"{label:24s}: ${c:>10,.0f}/day   ({c/baseline-1:+.0%})")

# Break-even: self-host the small model (spot GPU, Ch 33.8) vs the small-model API.
gpu_hr_spot, node_calls_per_sec = 0.90, 9.0
cap_day = node_calls_per_sec * 86400
selfhost_per_call = (24 * gpu_hr_spot) / cap_day          # cost/call at full utilisation
breakeven_util = selfhost_per_call / cost_call_S_c
print(f"\nself-host cost/call (full): ${selfhost_per_call:.6f}")
print(f"API cost/call (small)     : ${cost_call_S_c:.6f}")
print(f"break-even node utilisation: {breakeven_util:.1%} of one node's capacity")

baseline (all large)    : $   127,800/day   (+0%)
+ cache                 : $    83,965/day   (-34%)
+ cache + cascade       : $    25,406/day   (-80%)
+ all three levers      : $    24,458/day   (-81%)

self-host cost/call (full): $0.000028
API cost/call (small)     : $0.000358
break-even node utilisation: 7.7% of one node's capacity

Two results in Output 40.8.2 are worth dwelling on. First, the cascade, not the cache, is the dominant lever here, because the price gap between the small and large model is an order of magnitude and the small model handles three quarters of the uncached traffic; the cache and compression then trim what remains. Second, the break-even on self-hosting is extraordinarily low. A spot GPU node serving the small model costs a fraction of a cent per call at full utilization, far below the small-model API price, so any node kept busier than eight percent of its capacity is cheaper to own than to rent. This is the build-versus-buy crossover of Section 33.9 made concrete: APIs win on small and bursty volume where you cannot keep a node busy, and self-hosting wins decisively once volume is steady and large, with spot capacity from Section 33.8 widening the margin further.

6. Self-Hosting Versus the API at Fleet Scale Advanced

The break-even line in Output 40.8.2 generalizes into the build-versus-buy decision that every fleet eventually faces. An API charges a fixed price per token and scales perfectly to zero, so it is unbeatable for low or spiky volume where owned hardware would sit idle. Self-hosting converts that variable per-token cost into a fixed hourly cost for GPU nodes, which is only worthwhile when the nodes stay busy. The crossover is a utilization threshold: if a node serving the small model costs $H$ dollars per hour and serves a capacity of $\kappa$ calls per hour, its marginal cost per call at full load is $24H/(\kappa \cdot 24) = H/\kappa$, and self-hosting beats the API once

where $u$ is the fraction of capacity actually used. Output 40.8.2 puts that threshold near eight percent, which a steady fleet clears easily, so the strategic pattern is to keep the large frontier model on the API (it is hard to host, used rarely after the cascade, and changes often) while self-hosting the small workhorse model that absorbs the bulk of the traffic. Spot and preemptible capacity from Section 33.8 lowers $H$ further, and the quantized, batched serving of Chapter 22 raises $\kappa$, so both terms of the crossover move in the self-hoster's favor as volume grows. The full treatment of when to own the stack lives in Section 33.9; here the point is that the cost model tells you exactly where the line sits for your own numbers.