"I am one of two hundred trials, all of us convinced our learning rate is the special one. The scheduler does not share our optimism. Half of us will be quietly killed at epoch three, and the cruelest part is that it will probably be right."
A Trial That Plateaued Early
Chapter Overview
This is the final chapter of Part IV, and it steps back from any single training run to ask how the configuration that run uses gets chosen at all. Chapters 15 through 19 scaled one model across many machines and the previous chapter built the distributed infrastructure that trains a policy by reinforcement; in every case the hyperparameters arrived fixed, decided by someone before the cluster spun up. This chapter is about that someone's job, mechanized and distributed. The workload it studies is unlike any other in the book in one specific way: the units of work are independent, so the parallelism is free, and that freedom is exactly what makes the chapter's real subject visible. When parallelism costs nothing, the only thing left to engineer is allocation, and the chapter is a sustained study of how to spend a fixed compute budget across many candidate configurations when you cannot afford to finish them all.
Read in order, the eight sections take you from "the trials are independent, so the easy part is easy and the hard part is allocation" to "you can place configurations with Bayesian search, evaluate them at low fidelity, schedule compute with Hyperband or population-based training, run the whole thing across a cluster with early stopping, drive it from Ray Tune, and reason about whether the search pays for itself." The argument is cumulative: independence frees the parallelism, multi-fidelity evaluation makes early decisions possible, the schedulers act on those decisions, the distributed machinery runs them at scale, and the cost analysis tells you when to stop searching and start training.
Prerequisites
This chapter assumes the cluster and reliability machinery the earlier chapters built. From Chapter 18: Elastic and Fault-Tolerant Distributed Training you carry checkpointing and the ability to pause, resume, and migrate a run, because multi-fidelity scheduling depends on being able to stop a trial at a checkpoint and either resume it later or discard it cleanly. The chapter also leans forward on the cluster scheduling and resource-allocation ideas developed in Chapter 33: Cluster Infrastructure and Scheduling, since a trial scheduler is a domain-specific scheduler placing jobs on a shared, finite pool of accelerators; you can read this chapter first and Chapter 33 will generalize what you build here. Beyond that the chapter assumes comfortable Python, the basic machine-learning vocabulary of training, validation, learning curves, and overfitting, and a working sense of what a hyperparameter is and why its value matters, but no prior experience with an automated search system. Section 21.1 builds the allocation framing from the ground up before any specific algorithm appears.
Learning Objectives
- Explain why distributed hyperparameter search is embarrassingly parallel in its trials yet bottlenecked on allocation, framing the real problem as spending a finite compute budget rather than achieving parallelism.
- Compare grid search, random search, and Bayesian optimization as strategies for choosing where in the configuration space to evaluate, and reason about when each is appropriate.
- Articulate the multi-fidelity idea that a cheap, partial evaluation predicts a configuration's final quality, and identify the budget dimensions (epochs, data, resolution) it can exploit.
- Trace how successive halving allocates compute by repeatedly evaluating and pruning, and how Hyperband hedges across the trade-off between many cheap trials and few expensive ones.
- Describe population-based training as an evolutionary loop that exploits and explores a live population of configurations rather than discarding the losers.
- Design a distributed trial scheduler that makes early-stopping decisions across asynchronous workers reporting partial results at different rates, as in the ASHA algorithm.
- Map the chapter's abstractions onto Ray Tune and the surrounding AutoML ecosystem, recognizing what the library handles internally.
- Reason about cost-aware distributed experimentation, deciding when an additional search is worth its compute spend and how to bound the budget.
Chapter Roadmap
- 21.1 Why Search Is Embarrassingly Parallel, and Why That Is Not Enough Establishes that independent trials make the parallelism free, which is exactly why the real and binding problem is allocation: spending a finite compute budget across more configurations than the cluster can finish.
- 21.2 Grid, Random, and Bayesian Optimization Develops three strategies for choosing where in the configuration space to evaluate, from exhaustive grids through the surprising strength of random search to the surrogate-model placement of Bayesian optimization.
- 21.3 Multi-Fidelity Optimization Introduces the idea that reorganizes the rest of the chapter: a cheap, partial evaluation predicts a configuration's final quality, so compute can be reallocated away from doomed trials before they finish.
- 21.4 Successive Halving and Hyperband Turns the multi-fidelity idea into a schedule, repeatedly evaluating and pruning with successive halving, then hedging across the cheap-many versus expensive-few trade-off with Hyperband.
- 21.5 Population-Based Training Reframes search as an evolutionary loop over a live population, exploiting good configurations and exploring perturbations of them mid-flight rather than discarding the losers outright.
- 21.6 Distributed Trial Scheduling and Early Stopping Builds the scheduler that makes early-stopping decisions across asynchronous workers reporting partial results at different rates, the asynchronous successive-halving machinery that runs the search on a real cluster.
- 21.7 Ray Tune and the AutoML Ecosystem Maps the chapter's hand-built abstractions onto Ray Tune and the surrounding AutoML tools, showing how a few lines stand in for the schedulers, search algorithms, and distributed execution built by hand.
- 21.8 Cost-Aware Distributed Experimentation Closes the chapter and the part by treating compute as a budget to be justified, reasoning about when an additional search is worth its spend and how to bound the cost of finding a configuration.
What's Next?
This chapter closes Part IV. Across seven chapters the part took a single model and scaled its training across many machines, through data, model, sharded, and expert parallelism, elastic and fault-tolerant execution, frontier pre-training, distributed reinforcement learning, and finally the search over the configurations themselves; the recurring lesson was that the binding constraint is never one accelerator in isolation but the coordination that keeps a cluster of them productive. With that, the book pivots from building models to serving them. Chapter 22: Per-Node Inference Efficiency: A Prerequisite opens Part V, and it opens it deliberately on one machine. Before any fleet of replicas can be reasoned about, the unit economics of a single node, its quantization, its KV cache, its attention kernels, have to be understood, because distributed serving multiplies per-node behavior across the fleet and a fleet built on an inefficient node is an expensive mistake repeated many times. Read it next, and watch the scale-out story turn around: where Part IV asked how to spread one training run across many machines, Part V starts by asking what one machine actually costs to serve, then multiplies that answer across a distributed inference system.
Bibliography & Further Reading
Search Strategies and Bayesian Optimization
Bergstra, J., Bengio, Y. "Random Search for Hyper-Parameter Optimization." Journal of Machine Learning Research 13, 2012. jmlr.org/papers/v13/bergstra12a.html
The paper that showed random search beats grid search when only a few hyperparameters matter, the surprising baseline that frames the placement question of Section 21.2.
Snoek, J., Larochelle, H., Adams, R. P. "Practical Bayesian Optimization of Machine Learning Algorithms." arXiv:1206.2944, NeurIPS 2012. arxiv.org/abs/1206.2944
The work that made Gaussian-process Bayesian optimization a practical tool for tuning models, the surrogate-model placement strategy developed in Section 21.2.
Multi-Fidelity Scheduling
Jamieson, K., Talwalkar, A. "Non-stochastic Best Arm Identification and Hyperparameter Optimization (Successive Halving)." arXiv:1502.07943, AISTATS 2016. arxiv.org/abs/1502.07943
The paper that cast hyperparameter tuning as a best-arm problem and introduced successive halving, the pruning schedule at the core of Section 21.4.
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A. "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization." arXiv:1603.06560, JMLR 2018. arxiv.org/abs/1603.06560
The algorithm that hedges successive halving across the cheap-many versus expensive-few trade-off, the central scheduler studied in Section 21.4.
Falkner, S., Klein, A., Hutter, F. "BOHB: Robust and Efficient Hyperparameter Optimization at Scale." arXiv:1807.01774, ICML 2018. arxiv.org/abs/1807.01774
The method that combines Bayesian optimization with Hyperband, marrying the placement of Section 21.2 to the multi-fidelity scheduling of Section 21.4.
Distributed Scheduling and Population Methods
Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Hardt, M., Recht, B., Talwalkar, A. "A System for Massively Parallel Hyperparameter Tuning (ASHA)." arXiv:1810.05934, MLSys 2020. arxiv.org/abs/1810.05934
The asynchronous successive-halving algorithm that makes early-stopping decisions across a cluster of workers reporting at different rates, the direct reference for Section 21.6.
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., Kavukcuoglu, K. "Population Based Training of Neural Networks." arXiv:1711.09846, 2017. arxiv.org/abs/1711.09846
The work that introduced population-based training, the evolutionary exploit-and-explore loop over live configurations developed in Section 21.5.
Frameworks and AutoML Systems
Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., Stoica, I. "Tune: A Research Platform for Distributed Model Selection and Training." arXiv:1807.05118, 2018. arxiv.org/abs/1807.05118
The platform that packages distributed search, schedulers, and early stopping behind a small API, the primary framework mapped onto the chapter in Section 21.7.
Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M. "Optuna: A Next-generation Hyperparameter Optimization Framework." arXiv:1907.10902, KDD 2019. arxiv.org/abs/1907.10902
The define-by-run optimization framework with built-in pruning that complements Ray Tune in the ecosystem surveyed in Section 21.7.
Scaling Hyperparameters to Large Models
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., Gao, J. "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (muP)." arXiv:2203.03466, NeurIPS 2021. arxiv.org/abs/2203.03466
The result that tunes a small model and transfers the hyperparameters to a much larger one, a cost-aware alternative to searching at full scale that informs Section 21.8.