"I sorted the whole dataset by key so the reducers would agree. They did not thank me. They just took their partitions and got to work, as if the shuffle had cost nothing at all."
A Mapper That Did the Honest Thing and Shuffled
Before a model can be trained in parallel, the data that trains it has to be read, cleaned, transformed, and delivered in parallel; this part is about distributing the data axis, the layer beneath every later part. Part I named the six axes and gave the metrics that price them. Part II makes the first axis concrete: it shows how a dataset that no longer fits on one machine is partitioned across many, how a computation over that dataset is expressed so the partitions can be processed independently and recombined correctly, and how the resulting bytes are stored and streamed into the accelerators that consume them. The MapReduce shuffle introduced here is the same movement of keyed data that returns, scaled out, as the all-reduce of Part IV; the partitioning discipline established here is the foundation the training and serving parts assume from page one. Read this part as the answer to a question every later chapter takes for granted: where does the data come from, and how does it arrive fast enough to keep thousands of workers busy?
Part Overview
Distributed AI begins with distributed data. A foundation model trained on tens of terabytes of text, a recommender fed by a continuous river of click events, a retrieval index built over billions of documents: in every case the bottleneck that forces distribution first is not the model but the dataset. It outgrows the memory of one machine, then the disk, then the read bandwidth of one storage device, and each ceiling forces the same move that the rest of the book applies to computation: split the data across many machines, process the pieces where they live, and move only what must be moved. Part II is the systematic treatment of that move for the data axis, the layer on which Parts III through V build everything else.
The part proceeds from programming model to engine to storage to time. Chapter 6 opens with MapReduce, the model that made "partition, shuffle, recombine" a discipline anyone could program against, and the place where the shuffle that recurs throughout the book gets its first rigorous treatment. Chapter 7 moves to Spark and distributed DataFrames, the engine that kept MapReduce's fault tolerance while replacing its disk-bound rigidity with in-memory, lazily optimized pipelines that express most real feature engineering in a few lines. Chapter 8 turns to where the bytes actually live, distributed and object storage, columnar and sharded formats, and the data-loading path that feeds accelerators without starving them. Chapter 9 closes the part by removing the assumption that the dataset is finite and at rest, treating the unbounded stream and the online AI systems that learn and serve from data still in motion.
A word on why the data axis comes first. It is tempting to think of data processing as plumbing that precedes the interesting work of training and serving, but the same primitives govern both. The partitioning that splits a dataset across nodes is the partitioning that shards a model; the shuffle that regroups keyed records is the collective that regroups gradients; the freshness and latency budgets that a stream imposes on a feature pipeline are the budgets an online model inherits. Master the data layer and the later parts stop looking like separate subjects and start looking like the same partition-and-recombine move applied to heavier and heavier objects. That is the throughline this part installs and the rest of the book repays.
If you keep one thing from Part II, keep this: distributed AI is impossible without distributed data, and the partition-shuffle-recombine pattern that moves a dataset across machines is the same pattern that later moves gradients, shards, and activations. The four chapters are four refinements of one idea. Chapter 6 names the pattern in its purest form; Chapter 7 makes it fast and expressive; Chapter 8 grounds it in the storage and loading path that keeps accelerators fed; Chapter 9 lifts it from finite batches to unbounded streams. Read in order, they take you from a dataset sitting on disk to a live feature pipeline serving an online model, and they leave you with the data-layer vocabulary that Parts III through V assume on every page.
Part Roadmap
- 6 The MapReduce Model and Distributed Algorithms The programming model that made "partition, shuffle, recombine" a discipline, and the shuffle whose keyed regrouping returns, scaled out, as the all-reduce at the heart of every parallel method later in the book.
- 7 Spark and Distributed DataFrames The in-memory engine that kept MapReduce's fault tolerance while replacing its rigidity with lazily optimized DataFrame pipelines that express industrial feature engineering in a handful of lines.
- 8 Distributed Storage and Data Loading Where the bytes actually live: distributed and object storage, columnar and sharded formats, and the data-loading path that streams training data into accelerators fast enough to keep them busy.
- 9 Stream Processing and Online AI Removing the assumption that data is finite and at rest: the unbounded stream, windowing and watermarks, and the online AI systems that learn and serve from data still in motion.
Read the four chapters in order and you will carry the full data-layer vocabulary the rest of the book assumes: the MapReduce shuffle of Chapter 6 names the movement, Chapter 7 makes it fast, Chapter 8 grounds it in storage, and Chapter 9 sets it in motion. The thread to watch for begins in Chapter 6: the keyed shuffle you trace there by hand is the same regrouping of data that Chapter 15 performs on gradients, a connection the book returns to whenever a collective appears.
The deepest lesson of the data layer is that moving data is the dominant cost, and the art of distributed processing is arranging the computation so that data moves as little as possible. MapReduce makes this visible by isolating the one unavoidable data movement, the shuffle, and forcing you to pay for it explicitly; Spark makes it cheaper by keeping intermediate results in memory and letting an optimizer rewrite the pipeline to cut movement; distributed storage makes it predictable by laying bytes out in formats that read only the columns and shards a job needs; stream processing makes it continuous by moving small increments forever rather than huge batches occasionally. Every technique in this part is, at root, a different strategy for the same problem, get the work to the data instead of the data to the work, and that principle is exactly what Parts III through V apply to model parameters and activations once the data layer below them is solid.
What's Next?
Part II distributed the data axis: it showed how a dataset too large for one machine is partitioned, processed, stored, and streamed across many. Part III: Distributed Machine Learning takes the next axis, training, and turns the same partition-and-recombine move on the model and its optimization. The shuffle you learned to pay for here returns as the gradient all-reduce that synchronizes data-parallel workers; the partitioning that split a dataset here returns as the sharding that splits a parameter server's embedding tables; the freshness budgets a stream imposed here return as the staleness a distributed optimizer must tolerate. Read Part III next, and the data-layer primitives you just mastered will reappear as the machinery of distributed learning, applied to a heavier object but governed by the same arithmetic of communication and failure.