Cost and Performance Analysis

"They gave me sixteen nodes and asked why it was slower than eight. I pointed at the one stage that ate the afternoon, and the room went quiet."

A Profiler, Pointing at the One Stage That Ate the Afternoon

Section 41.6 told you which metrics to collect: wall-clock per run, throughput, speedup, parallel efficiency, and dollar cost. This section is about reading those metrics as evidence. A distributed AI system that achieves a 3.9-fold speedup on eight nodes has not earned the number until you can say where the missing 4.1-fold went, and a system that costs two dollars per run has not earned a deployment decision until you can say what a one-dollar run would have cost you in latency. Performance analysis is the discipline of attributing the gap between the speedup you hoped for and the speedup you measured, and cost analysis is the discipline of pricing that gap. Both rest on the performance models of Chapter 3 and the communication-cost accounting of Chapter 4; here we put them to work on a system you built.

1. Where the Time Actually Goes Beginner

The first question of any performance analysis is not "how fast is it?" but "what is it doing while the clock runs?" A single-node program spends its wall-clock in compute and in I/O, and a profiler that attributes time to functions is enough to find the slow part. A distributed program adds two categories that do not exist on one machine, and they are usually where the surprises hide. The wall-clock of a run on $p$ nodes decomposes, to a first approximation, into time spent computing, time spent moving data between machines, and time spent waiting,

where $T_{\text{idle}}$ collects every form of doing nothing useful: a fast worker blocked at a barrier for a slow one (the straggler tax of Chapter 3), a node stalled on a disk or object-store read (the I/O of Chapter 8), or a pipeline bubble while one stage waits for another. The decomposition matters because each term has a different remedy. Compute-bound time wants a faster kernel or a bigger batch; communication-bound time wants a better collective schedule or gradient compression; idle time wants load balancing or overlap. Attributing your wall-clock to the wrong term sends you optimizing the part that was never the bottleneck.

Figure 41.7.1 shows why the average is a misleading summary. Every worker finishes its own compute at a different moment, but the synchronous all-reduce of Chapter 15 forces them to a common barrier, so the step takes as long as the slowest worker plus the collective. The grey idle bands are pure waste: capacity you rented and did not use. A profiler is the instrument that exposes those bands. PyTorch's profiler, Nsight Systems, and Ray's timeline view all record per-stage spans you can sum into the three buckets of the decomposition; the library shortcut below shows the smallest version that already separates compute from communication.

import torch
from torch.profiler import profile, ProfilerActivity, schedule

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule(wait=1, warmup=1, active=3),   # skip startup, profile 3 steps
    record_shapes=True,
) as prof:
    for step in range(5):
        train_one_step()          # forward + backward + DDP all-reduce
        prof.step()

# Communication shows up as nccl:all_reduce / nccl:all_gather rows; compute as
# the matmul/attention kernels. Sort by total CUDA time to find the dominant bucket.
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=12))
prof.export_chrome_trace("trace.json")   # open in chrome://tracing or Perfetto

2. Reading the Scaling Curve and Finding the Knee Intermediate

Once the per-node breakdown is in hand, the across-node view is the scaling curve: speedup $S_p = T_1 / T_p$ plotted against node count $p$, and its companion, parallel efficiency $E_p = S_p / p$. Ideal scaling is the line $S_p = p$, which no real system reaches. What you actually measure is a curve that tracks the ideal line at small $p$, peels away from it as communication and idle time grow, flattens at a maximum, and on many systems bends back down once added nodes cost more in coordination than they contribute in compute. The shape is not noise; it is the signature of the three terms in the decomposition fighting the one term that shrinks. Compute time falls roughly as $T_{\text{comp}}(p) \approx T_{\text{comp}}(1)/p$, but communication and idle time grow with $p$, so the total has a minimum.

The most decision-relevant feature of the curve is its knee: the node count past which efficiency drops sharply, where each new node buys far less than the one before it. Amdahl's law, from Chapter 3, gives the ceiling that creates the knee. If a fraction $f$ of the work is irreducibly serial, the best speedup any number of nodes can reach is

so a system with even five percent serial work cannot exceed a twenty-fold speedup no matter how many nodes you rent. The knee is where you collide with that asymptote, or, on a synchronous system, where communication and straggler idle time overtake the shrinking compute. Attributing the knee is the heart of the analysis: read it off the curve, then look back at the per-node breakdown of Figure 41.7.1 to say whether it is Amdahl serial fraction, a communication wall, or load imbalance that bent the line. Honest reporting names the cause, not just the node count.

3. Pricing the Speedup: Cost and Diminishing Returns Intermediate

Speedup is bought with money, and the conversion is direct. If a run on $p$ nodes takes wall-clock $T_p$ at a price of $c$ dollars per node-hour, the dollar cost of that run is

Because $T_p$ falls more slowly than $1/p$ once you pass the knee, the product $p \cdot T_p$, and therefore the cost, rises with scale even while the run gets faster. You are paying a strictly increasing amount for a speedup that grows sublinearly and eventually not at all. The decision-grade quantity is therefore not speedup alone but marginal speedup per dollar: how much additional speedup the next increment of nodes buys for its additional cost,

This ratio collapses toward zero, and can go negative, exactly at the knee. A capstone that reports it has converted raw timings into an economic argument: there is a scale below which more nodes are an obvious bargain, a scale above which they are an obvious waste, and a narrow band in between where the right answer depends on whether you are buying speed or buying frugality. The cost-optimal node count $p^{\star}$ minimizes $\text{Cost}(p)$; the latency-optimal count maximizes $S_p$; and these are rarely the same number, which is the whole tension a deployment decision must resolve.

The runnable demo below carries out exactly this analysis end to end. It takes a measured time breakdown per scale and a node price, computes the wall-clock, speedup, efficiency, and dollar cost at each scale, then reports the marginal speedup per dollar across the ladder and names the cost-optimal node count.

import numpy as np

# A measured time breakdown per scale: for each node count p, the wall-clock
# seconds spent in compute, communication, and idle/straggler waiting for ONE run.
# These come from a profiler (Ch 3/4 cost model in practice). T_p = comp + comm + idle.
breakdown = {
    1:  {"comp": 800.0, "comm":   0.0, "idle":   0.0},
    2:  {"comp": 400.0, "comm":  18.0, "idle":  12.0},
    4:  {"comp": 200.0, "comm":  34.0, "idle":  26.0},
    8:  {"comp": 100.0, "comm":  58.0, "idle":  48.0},
    16: {"comp":  50.0, "comm":  95.0, "idle":  90.0},
}
price_per_node_hour = 3.20   # on-demand dollars per node-hour

T1 = sum(breakdown[1].values())          # single-node wall-clock, the speedup baseline
rows = []
for p, b in breakdown.items():
    Tp = b["comp"] + b["comm"] + b["idle"]
    speedup = T1 / Tp
    efficiency = speedup / p
    cost = price_per_node_hour * p * (Tp / 3600.0)   # $ = price * nodes * hours
    rows.append({"p": p, "Tp": Tp, "speedup": speedup, "eff": efficiency, "cost": cost})

print(f"baseline T1 = {T1:.1f} s   price = ${price_per_node_hour:.2f}/node-hour\n")
hdr = f"{'p':>3} {'T_p(s)':>8} {'speedup':>8} {'eff':>6} {'cost($)':>8} {'comm%':>6} {'idle%':>6}"
print(hdr)
print("-" * len(hdr))
for r, (p, b) in zip(rows, breakdown.items()):
    commp = 100.0 * b["comm"] / r["Tp"]
    idlep = 100.0 * b["idle"] / r["Tp"]
    print(f"{r['p']:>3} {r['Tp']:>8.1f} {r['speedup']:>8.2f} {r['eff']:>6.2f} "
          f"{r['cost']:>8.3f} {commp:>6.1f} {idlep:>6.1f}")

# Marginal speedup-per-dollar: how much extra speedup each extra dollar buys as we
# step up the scale ladder. Diminishing returns show up as this number collapsing.
print("\nmarginal analysis (each step up the ladder):")
print(f"{'p_lo->p_hi':>12} {'d_speedup':>10} {'d_cost($)':>10} {'speedup/$':>10}")
for lo, hi in zip(rows[:-1], rows[1:]):
    d_speed = hi["speedup"] - lo["speedup"]
    d_cost = hi["cost"] - lo["cost"]
    ratio = d_speed / d_cost if d_cost > 0 else float("inf")
    print(f"{str(lo['p'])+'->'+str(hi['p']):>12} {d_speed:>10.2f} "
          f"{d_cost:>10.3f} {ratio:>10.2f}")

# Cost-optimal node count: the scale with the lowest dollar cost per run.
best = min(rows, key=lambda r: r["cost"])
fastest = max(rows, key=lambda r: r["speedup"])
print(f"\ncost-optimal scale : p = {best['p']}  (${best['cost']:.3f}/run, "
      f"speedup {best['speedup']:.2f}x)")
print(f"fastest scale      : p = {fastest['p']}  (${fastest['cost']:.3f}/run, "
      f"speedup {fastest['speedup']:.2f}x)")
print(f"paying for speed   : {fastest['cost']/best['cost']:.2f}x the cost "
      f"to go {fastest['speedup']/best['speedup']:.2f}x faster")

baseline T1 = 800.0 s   price = $3.20/node-hour

  p   T_p(s)  speedup    eff  cost($)  comm%  idle%
---------------------------------------------------
  1    800.0     1.00   1.00    0.711    0.0    0.0
  2    430.0     1.86   0.93    0.764    4.2    2.8
  4    260.0     3.08   0.77    0.924   13.1   10.0
  8    206.0     3.88   0.49    1.465   28.2   23.3
 16    235.0     3.40   0.21    3.342   40.4   38.3

marginal analysis (each step up the ladder):
  p_lo->p_hi  d_speedup  d_cost($)  speedup/$
        1->2       0.86      0.053      16.13
        2->4       1.22      0.160       7.60
        4->8       0.81      0.540       1.49
       8->16      -0.48      1.877      -0.26

cost-optimal scale : p = 1  ($0.711/run, speedup 1.00x)
fastest scale      : p = 8  ($1.465/run, speedup 3.88x)
paying for speed   : 2.06x the cost to go 3.88x faster

4. Total Cost of Ownership: Cloud, Local, and Spot Advanced

The per-run cost of Code 41.7.2 is the right unit for a single job, but a deployed system runs many jobs over months, and the deployment decision turns on total cost of ownership rather than the price of one run. The cluster-economics treatment of Chapter 33 draws the three regimes a capstone should weigh. On-demand cloud charges the full $c$ per node-hour with zero commitment, which is correct for bursty or one-off work and is exactly what Code 41.7.2 prices. Owned local hardware front-loads a capital cost and then runs near the marginal cost of electricity, so it wins only above a utilization threshold: a cluster busy enough that the amortized purchase price per node-hour drops below the cloud rate. Spot and preemptible instances cut the hourly price by often two-thirds to four-fifths in exchange for the risk of preemption, which is acceptable precisely when the elastic and checkpointing machinery of Chapter 18 lets a job survive losing a node mid-run.

A capstone cost analysis is honest only when it states which regime its numbers assume and re-prices the cost-optimal scale under the alternatives. The same eight-node run that costs one dollar forty-seven on-demand might cost forty cents on spot, shifting $p^{\star}$ outward because cheaper nodes change where the marginal-dollar argument crosses zero, or it might cost effectively nothing on an already-owned cluster you are sharing, in which case wall-clock alone decides. The discipline is to make the assumption explicit, because a speedup curve priced under the wrong regime recommends the wrong scale. The breakdown of Figure 41.7.1 is invariant across regimes; the dollar figures are not.

5. An Analysis Template: Measure, Attribute, Decide Beginner

The preceding sections compose into a fixed procedure you can apply to any distributed AI system, and reusing the same three steps keeps an analysis honest and comparable. Table 41.7.1 lays out the template: a measure phase that produces the per-node breakdown and the scaling curve, an attribute phase that names the cause of every gap from ideal, and a decide phase that converts the attributed curve into a recommended operating point with its cost stated under a named regime. The output of this template is what Section 41.9 folds into the capstone report.

The template's discipline is that the decide phase may never run ahead of the attribute phase. A recommended node count without a named reason for the curve's shape is a guess dressed as a result, and a reviewer will ask the question the analysis skipped. Run the three phases in order, report the breakdown that motivated each conclusion, and the recommendation defends itself. Section 41.6 supplied the metrics that feed phase one; Section 41.9 consumes the phase-three artifact into the written report.