"I asked the cluster for sixty-four GPUs that could talk to each other quickly. It gave me sixty-four GPUs, eventually, on the understanding that two of them were in another building and one was already someone else's."
A Training Job Waiting in the Queue
Chapter Overview
Until now this book has lived one level above the machine. A data-parallel step in Chapter 15 assumed the workers existed, were connected, and stayed alive long enough to finish an epoch. The serving fleet of Chapter 24 assumed the replicas had somewhere to run. Part VII descends to the substrate those assumptions stand on. A cluster is hardware (compute nodes, accelerators, a network fabric, storage) plus a control plane that turns a stream of submitted work into a sequence of placements on that hardware. The control plane is the scheduler, and scheduling is the central act of this chapter: given who wants to run and what is available, decide who runs where, for how long, and at what priority. For ordinary batch jobs that is a packing problem. For AI workloads it is something sharper, because AI imposes demands that generic schedulers were never designed to meet.
A word on why this matters even if you never administer a cluster yourself. The placement decisions in this chapter set the constants in every performance model the book has built. The communication cost of Chapter 3 depends on whether the scheduler put your workers one hop apart or three; the failure rate that drives Chapter 18 depends on whether you bought reliable on-demand capacity or cheap preemptible capacity; the throughput of the serving fleet depends on how tightly the scheduler packs replicas onto accelerators. Reading a system's scheduler is reading the boundary conditions of its physics. This chapter gives you the language to do that.
Prerequisites
This chapter sits near the end of the book and draws the algorithmic threads down onto hardware, so it assumes the parts that defined those threads. From Chapter 4 it assumes the collective communication primitives (all-reduce, all-gather, all-to-all) and the idea that their cost depends on network topology, which is exactly what collective-aware placement in Section 33.5 optimizes. From Chapter 15 and Chapter 16 it assumes data, model, and sharded parallelism, the gang of tightly coupled workers that gang scheduling exists to serve. From Chapter 18 it assumes elastic and fault-tolerant training, the checkpointing and recovery machinery that spot scheduling in Section 33.8 leans on. From Chapters 22 through 24 it assumes the per-node inference economics and fleet sizing that make GPU packing and sharing a cost question rather than an abstract one. Readers comfortable with those four threads can read this chapter as the place where they finally touch the metal.
Learning Objectives
- Decompose an AI cluster into hardware (compute nodes, accelerators, network fabric, storage) and a control plane, and explain what a scheduler does with the two.
- Distinguish CPUs, GPUs, and TPUs by memory and interconnect, and read an accelerator instance type for the ceilings it imposes on a workload.
- Explain how containers and Kubernetes make a heterogeneous cluster addressable, and place a workload on a batch scheduler (Slurm, Kubernetes batch, or Volcano).
- Justify gang scheduling for tightly coupled training and describe collective-aware placement that keeps the all-reduce of Chapter 4 cheap.
- Choose among MIG, MPS, and time-slicing to share one accelerator across several jobs, and reason about the isolation each provides.
- Model the cost and risk of spot and preemptible scheduling using the checkpoint-interval mathematics of Chapter 18, and weigh self-run clusters against managed platforms.
If you keep one thing from this chapter, keep this: scheduling is where a distributed algorithm meets a finite, contended, failure-prone pile of hardware, and the placement it produces sets the constants in every performance model you have built. A training gang wants all its workers at once and close together; an inference fleet wants tight packing and cheap fractional accelerators; a cost-conscious job wants spot capacity and a checkpoint cadence that survives preemption. These pulls conflict, and the scheduler is where they are reconciled. Read forward, the chapter is a tour of the schedulers (Slurm, Volcano, Kubernetes, Ray) and the sharing and cost mechanisms (MIG, MPS, spot) that resolve those pulls in different ways. Read as a question, it is a single checklist you apply to any cluster: what is the hardware, what is the control plane, and what does its placement cost the algorithm running on top? The roadmap below walks the nine sections that build that checklist.
Chapter Roadmap
- 33.1 Anatomy of an AI Cluster The parts beneath every workload: compute nodes, accelerators, the network fabric, storage, and the control plane that turns submitted work into placements, with the shared vocabulary the rest of the chapter reuses.
- 33.2 Compute: CPUs, GPUs, TPUs, and Accelerator Instances The compute itself, read for the ceilings it imposes: device memory, interconnect bandwidth, and the accelerator instance types whose shape every later section schedules around.
- 33.3 Containers and Kubernetes for AI The packaging and orchestration layer that makes a heterogeneous cluster addressable: containers for reproducible environments and Kubernetes for declaring, placing, and reconciling AI workloads.
- 33.4 Batch Schedulers: Slurm, Kubernetes Batch, and Volcano The queue-and-place engines for long-running jobs, from the HPC heritage of Slurm to the Kubernetes-native batch and Volcano schedulers that bring AI-aware policies to the cloud-native world.
- 33.5 Gang Scheduling and Collective-Aware Placement The demand that makes AI scheduling distinctive: a training job is one gang that must start together or not at all, placed where the collective communication of Chapter 4 stays cheap.
- 33.6 Multi-Tenant GPU Sharing: MIG, MPS, and Time-Slicing Covers MIG, MPS, and time-slicing GPU sharing, then introduces the 2025 Kubernetes layer: Dynamic Resource Allocation (DRA, GA in K8s 1.34) for structured fractional GPU claims, Kueue for quota and cohort admission control, and NVIDIA KAI Scheduler for topology-aware gang scheduling on NVLink domains.
- 33.7 Ray Clusters and the Object Store A Python-native cluster framework with a distributed object store at its core, carrying much of the modern training, tuning, and serving stack as an alternative to Kubernetes and Slurm.
- 33.8 Spot and Preemptible Scheduling for Cost Optimization The cheapest capacity in the cloud comes with the right to take it back; the discount is real, and paying for it correctly turns the checkpoint-interval mathematics of Chapter 18 into a scheduling decision.
- 33.9 Managed Platforms: Databricks, SageMaker, and Vertex AI The chapter's closing trade: rent a cluster that hides the scheduler entirely, or run your own and keep the control. What each managed platform automates, and what it costs to give up.
What's Next?
This chapter kept the work inside the datacenter, where the network is fast, the power is plentiful, and the scheduler owns a contiguous fabric of machines. Chapter 34: Edge, Fog, and On-Device Distributed AI pushes the infrastructure outward from that warm center to the periphery, where the machines are phones, sensors, vehicles, and gateways, the network is intermittent, and no single scheduler owns anything. The gang scheduling and topology-aware placement of this chapter assumed a cluster you control; the edge withdraws that assumption and asks how distributed AI behaves when compute is scattered across thousands of weak, unreliable, geographically spread devices. The placement problem does not vanish, it inverts, and the federated and decentralized ideas of Chapter 14 return as the way to cope. Read it next to follow the infrastructure from the rack to the road.
The 2026 cluster-platform frontier is moving from "request N GPUs" to "request an accelerator topology with named properties." Kubernetes Dynamic Resource Allocation gives devices the same declarative flavor that persistent volumes gave storage: a job can ask for a class of accelerators, topology constraints, MIG slices, or a multi-device allocation rather than a bare integer such as nvidia.com/gpu: 8. On top of that substrate, Kueue manages admission, cohorts, and quota borrowing, while NVIDIA KAI Scheduler focuses on topology-aware gang placement and fractional GPU sharing. For a distributed training job, these are not administrative details. They decide whether the all-reduce graph of Chapter 4 lands inside a fast NVLink island, across a rack fabric, or through a slow path that turns the scheduler into the bottleneck.
A placement decision changes the effective algorithm by changing the communication graph. The same tensor-parallel plan from Chapter 16 can be viable on a tightly connected accelerator island and untenable when its ranks are scattered. Modern AI platforms therefore schedule devices, memory, network locality, security posture, and preemption risk together. The cluster is no longer a neutral box that runs the algorithm; it is one of the algorithm's parameters.
Bibliography & Further Reading
Foundational Papers
Yoo, A. B., Jette, M. A., Grondona, M. "SLURM: Simple Linux Utility for Resource Management." Job Scheduling Strategies for Parallel Processing (JSSPP), 2003. slurm.schedmd.com
The original paper for the batch scheduler that runs most of the world's HPC and a large share of its training clusters; the queue-and-place model that Section 33.4 starts from.
Verma, A., Pedrosa, L., Korupolu, M., et al. "Large-scale cluster management at Google with Borg." EuroSys 2015. research.google
The system whose ideas became Kubernetes: priority, preemption, and bin-packing at warehouse scale. The intellectual ancestor of the control plane in Sections 33.3 and 33.4.
Moritz, P., Nishihara, R., Wang, S., et al. "Ray: A Distributed Framework for Emerging AI Applications." OSDI 2018. usenix.org
Introduces Ray and the distributed object store at its center; the design Section 33.7 develops as the Python-native alternative to Kubernetes and Slurm.
Daly, J. T. "A higher order estimate of the optimum checkpoint interval for restart dumps." Future Generation Computer Systems 22(3), 2006. sciencedirect.com
The refined formula for how often to checkpoint given a failure rate and checkpoint cost; the mathematics Section 33.8 borrows to schedule against preemption.
Young, J. W. "A first order approximation to the optimum checkpoint interval." Communications of the ACM 17(9), 1974. dl.acm.org
The original square-root rule relating optimal checkpoint interval to mean time between failures; the half-century-old result that still sizes spot-training cadence in Section 33.8.
Tools & Libraries
Kubernetes Documentation: scheduling, preemption, and eviction. kubernetes.io/docs
The reference for the cloud-native control plane: how the default scheduler places pods, and how priority and preemption work, the foundation Section 33.3 builds on.
Volcano: a Kubernetes-native batch scheduling system (CNCF). volcano.sh
The kube-batch successor that brings gang scheduling, fair-share queues, and topology awareness to Kubernetes; the AI-aware batch scheduler of Sections 33.4 and 33.5.
NVIDIA Multi-Instance GPU (MIG) User Guide. docs.nvidia.com
The official guide to partitioning one accelerator into hardware-isolated instances; the strong-isolation end of the GPU-sharing spectrum in Section 33.6.
NVIDIA Multi-Process Service (MPS) Documentation. docs.nvidia.com
The official reference for sharing a GPU across processes through a single context; the higher-utilization, weaker-isolation counterpart to MIG in Section 33.6.
Ray Documentation: clusters, autoscaling, and the object store. docs.ray.io
The practical entry point for standing up and autoscaling a Ray cluster; the toolkit behind Section 33.7 and much of the training and serving stack elsewhere in the book.
SkyPilot: run AI on any cloud. docs.skypilot.co
An open framework for launching jobs across clouds and chasing spot capacity automatically; a concrete realization of the cost-driven spot scheduling of Section 33.8.
Platforms & Documentation
Amazon SageMaker Documentation: training and inference. docs.aws.amazon.com
The managed-platform reference for AWS; a concrete instance of the hide-the-scheduler trade weighed in Section 33.9.
Google Cloud Vertex AI Documentation. cloud.google.com
Google Cloud's managed AI platform, with TPU access and pipeline orchestration; the second of the three managed platforms compared in Section 33.9.
Databricks Documentation: clusters and jobs. docs.databricks.com
The Spark-rooted lakehouse platform with managed clusters and job scheduling; the data-and-ML managed option in Section 33.9, tying back to the Spark of Part II.