"They benchmarked me on eight GPUs, drew a straight line through two points, and promised their investors linear scaling to a thousand. I am the ninth GPU. I would like a word."
A Benchmark That Was Run Exactly Once
Chapter Overview
Evaluation is where distributed AI stops being a design exercise and becomes an empirical science. A system that runs is not the same as a system that scales, and a system that scales on a slide is not the same as one that scales on a cluster you do not own. This chapter assembles the measurement discipline that tells the difference: the metrics that matter for distributed work, the way those metrics behave as you add machines, and the methodology that keeps a measurement from lying to the person who took it.
The chapter opens by arguing that distribution needs its own evaluation discipline, because the failure modes of a distributed system, stragglers, communication overhead, contention, partial failure, are invisible to single-node metrics. It then builds the core quantities one layer at a time: the speedup and efficiency curves that say how well work parallelizes, the throughput, goodput, and tail-latency targets that say how a serving system behaves under load, the communication-to-computation ratio that explains why a curve bends, and the cost, utilization, and energy ledgers that say what the run actually consumed. It closes with the methodology that ties all of it together: how to benchmark without fooling yourself, and how to make a measurement reproducible on a cluster.
Read in order, the seven sections take you from "why single-node metrics mislead" to a working evaluation toolkit that every later part of the book leans on, whether you are studying a MapReduce job in Chapter 6, a data-parallel training run in Chapter 15, or an LLM serving fleet in Chapter 24.
Prerequisites
This chapter assumes you have read Chapter 1: What Is Scale-Out AI?, Chapter 2: Distributed Systems Concepts for AI, Chapter 3: Scalability and Performance Models, and Chapter 4: Communication Primitives for Distributed Training. From Chapter 1 you carry the operational metrics and the axes of distribution that this chapter learns to measure. From Chapter 2 you carry the failure vocabulary, stragglers, partial failure, contention, that explains why distributed evaluation is hard. The load-bearing prerequisite is Chapter 3: the speedup and efficiency definitions of Chapter 3, together with Amdahl's and Gustafson's laws, are the curves this chapter teaches you to measure rather than merely derive, and the alpha-beta communication cost model of Chapter 4 is what the communication-to-computation ratio of Section 5.4 turns into an observable quantity. No statistics beyond means, variance, and percentiles is assumed; the methodology section introduces the rest where it is needed. The mathematical background the book assumes is refreshed in Appendix A: Mathematical Background.
Learning Objectives
- Explain why single-node performance metrics mislead for distributed systems, and name the failure modes that distributed evaluation must surface.
- Measure and interpret speedup, efficiency, and scalability curves, distinguishing strong scaling from weak scaling and identifying a missing or unfair baseline.
- Define throughput, goodput, and tail latency, and reason about service-level objectives stated as percentiles rather than averages.
- Compute the communication-to-computation ratio for a workload and use it to predict where a scaling curve bends.
- Account for the cost, utilization, and energy of a distributed run, including model FLOPs utilization and dollars-per-result.
- Design a benchmark that avoids the common pitfalls (warmup, variance, cherry-picked baselines) and document it so another team can reproduce the measurement on a different cluster.
Chapter Roadmap
- 5.1 Why Distribution Needs Its Own Evaluation Discipline Argues that the failure modes of distributed systems, stragglers, communication overhead, contention, and partial failure, are invisible to single-node metrics and demand their own measurement vocabulary.
- 5.2 Speedup, Efficiency, and Scalability Curves Turns the speedup and efficiency definitions of Chapter 3 into measured curves, distinguishing strong from weak scaling and exposing the missing-baseline trap that flatters every scaling plot.
- 5.3 Throughput, Goodput, and Tail Latency / SLOs Separates the work a system does from the useful work it does, and states serving targets as percentile service-level objectives rather than averages that hide the slow tail.
- 5.4 Communication-to-Computation Ratio Makes the alpha-beta cost of Chapter 4 an observable quantity, the ratio that predicts where a scaling curve bends and how much of each step the network steals.
- 5.5 Cost, Utilization, and Energy Accounting Prices a run in dollars and joules, introducing model FLOPs utilization, hardware utilization, and the cost-per-result that turns a fast system into an affordable one.
- 5.6 Benchmarking Methodology and Pitfalls The discipline of measuring without fooling yourself: warmup, variance, confidence intervals, fair baselines, and the cherry-picking that makes a benchmark a marketing artifact.
- 5.7 Reproducible Measurement on Clusters How to pin the environment, control for noisy neighbors, and document a run so another team gets the same answer on different hardware, the standard the rest of the book holds itself to.
What's Next?
This chapter closes Part I. You now hold the foundations of distributed AI: the axes of distribution, the systems concepts, the performance models, the communication primitives, and the evaluation discipline that measures whether any of it works. Chapter 6: The MapReduce Model and Distributed Algorithms opens Part II and puts the foundations to work on the first real distributed workload: processing data at a scale no single machine can hold. The speedup and efficiency curves you just learned to measure become the way you judge a MapReduce job; the shuffle at the heart of MapReduce is the same all-reduce you met in Chapter 4 wearing a data-processing disguise. Read it next, and the foundations of Part I become the engine of Part II.
Bibliography & Further Reading
Benchmarks and Benchmarking Methodology
Mattson, P., Cheng, C., Coleman, C., et al. "MLPerf Training Benchmark." arXiv:1910.01500, 2019. arxiv.org/abs/1910.01500
The community training benchmark that defines fair time-to-accuracy measurement across hardware and frameworks; the reference standard behind the methodology of Sections 5.2 and 5.6.
Reddi, V. J., Cheng, C., Kanter, D., et al. "MLPerf Inference Benchmark." arXiv:1911.02549, 2019. arxiv.org/abs/1911.02549
The inference companion to MLPerf Training, defining latency-bounded and throughput scenarios; the grounding for the throughput and tail-latency targets of Section 5.3.
Hoefler, T., Belli, R. "Scientific Benchmarking of Parallel Computing Systems." International Conference for High Performance Computing (SC), 2015. htor.inf.ethz.ch
The twelve rules for reporting parallel-computing performance with confidence intervals and reproducibility; the backbone of the pitfalls and methodology of Sections 5.6 and 5.7.
MLCommons. "MLPerf Benchmarks and Results." mlcommons.org
The consortium that maintains the MLPerf suites and publishes audited, reproducible results; the living source of the fair-comparison rules Section 5.6 teaches you to apply.
Latency, Throughput, and Utilization
Dean, J., Barroso, L. A. "The Tail at Scale." Communications of the ACM 56(2), 2013. cacm.acm.org
The paper that made tail latency a first-class metric and explained why averages lie at fleet scale; the conceptual heart of the SLO discussion in Section 5.3.
Chowdhery, A., Narang, S., Devlin, J., et al. "PaLM: Scaling Language Modeling with Pathways." arXiv:2204.02311, 2022. arxiv.org/abs/2204.02311
Introduces model FLOPs utilization (MFU) as a hardware-independent training-efficiency metric; the definition Section 5.5 uses to price a run in useful compute.
Cost and Energy Accounting
Patterson, D., Gonzalez, J., Le, Q., et al. "Carbon Emissions and Large Neural Network Training." arXiv:2104.10350, 2021. arxiv.org/abs/2104.10350
The reference accounting of energy and carbon for large training runs, with the factors that dominate the joule ledger; the foundation of the energy section of Section 5.5.
Strubell, E., Ganesh, A., McCallum, A. "Energy and Policy Considerations for Deep Learning in NLP." arXiv:1906.02243, 2019. arxiv.org/abs/1906.02243
The early estimate of the energy and dollar cost of training and tuning large NLP models; the motivation for treating cost-per-result as a first-class metric in Section 5.5.
Profiling Tools
PyTorch Profiler Documentation. pytorch.org
The in-framework profiler that attributes time to compute and communication kernels; the tool Section 5.4 uses to measure the communication-to-computation ratio in practice.
NVIDIA Nsight Systems Documentation. docs.nvidia.com
The system-wide timeline profiler that exposes GPU utilization, kernel overlap, and communication gaps; the instrument behind the utilization accounting of Sections 5.4 and 5.5.