"They told me I was one model. I have since learned that I am a data pipeline that crawled half the web, a thousand GPUs that agreed to stay synchronized for six weeks, and a learning-rate schedule nobody was allowed to touch. The weights are just where we all finally agreed."
A Foundation Model, Reflecting on Its Own Pre-Training
Chapter Overview
This is the fifth chapter of Part IV, and it is the one that stops teaching a technique and starts building a system. Chapters 15 through 18 each gave you one axis: data parallelism, model and pipeline and sharded parallelism, expert parallelism, and the elasticity and fault tolerance that keep a long job alive. This chapter assumes you hold all of them and asks the harder question of how they combine, with the data pipeline of Part II underneath, into the single most ambitious distributed-systems artifact most engineers will ever touch. The binding constraint is no longer any one resource; it is the orchestration that keeps every piece moving in lockstep, because at this scale the bottleneck is whichever stage is currently failing to keep up, and the cost of getting it wrong is measured in weeks of wasted accelerator time and millions of dollars.
Read in order, the nine sections take you from "a foundation model is the output of a distributed system, not a thing you simply download" to "you can size a run from a compute budget, build the petabyte pipeline that feeds it, orchestrate the multi-thousand-GPU job that trains it, align the result into something useful, and account for what it cost." The argument is cumulative and integrative: the scaling laws size the run, the data pipeline feeds it, the orchestration composes every axis of this part into one job, alignment turns the raw model into an assistant, and the cost accounting closes the loop on what spending a data center for a single model actually means.
Prerequisites
This chapter is the capstone of Part IV and assumes you have read the rest of it. From Chapter 16: Model, Pipeline, and Sharded Parallelism you carry the sharded optimizer and the tensor and pipeline partitioning that let a model larger than any single device fit and train; from Chapter 18: Elastic and Fault-Tolerant Distributed Training you carry the checkpointing, deterministic restart, and elasticity that keep a multi-week run alive on hardware that will not stay healthy for its full duration. The data half of this chapter assumes the distributed data processing of Part II, and in particular the Chapter 6: The MapReduce Model and Distributed Algorithms shuffle, because the trillion-token deduplication and cleaning pipeline is a distributed-algorithms problem before it is a machine-learning one. The chapter also leans on the all-reduce of Chapter 4 and the data-parallel step of Chapter 15. It assumes comfortable Python and PyTorch and a working memory of how the parallelism axes of this part compose. No prior experience running a frontier-scale job is required; Section 19.1 builds the systems framing from the ground up before any single component is detailed.
Learning Objectives
- Reframe a foundation model as the output of an end-to-end distributed system, and identify the components that span data processing, optimization, parallelism, and fault tolerance.
- Apply scaling laws to turn a fixed compute budget into a compute-optimal choice of parameter count and training-token count.
- Architect a distributed dataset-construction pipeline that crawls, filters, and shards a corpus of trillions of tokens across a cluster.
- Design distributed deduplication and quality-filtering passes that scrub near-duplicate and low-value documents at corpus scale using shuffle-based algorithms.
- Build a tokenization pipeline that trains and applies a vocabulary across a dataset too large to process on a single machine.
- Orchestrate a pre-training run that composes data, tensor, pipeline, sharded, and expert parallelism with checkpointing, stability engineering, and loss-spike recovery.
- Distribute fine-tuning across machines, and place parameter-efficient methods correctly as a per-node enabler within the distributed picture.
- Explain distributed alignment as a systems problem, covering supervised fine-tuning, reinforcement learning from human feedback, and direct preference optimization.
- Account for the energy, dollar cost, and carbon footprint of a frontier-scale run, and reason about responsible-scaling tradeoffs.
Chapter Roadmap
- 19.1 Foundation Models as Distributed Systems Reframes a foundation model as the output of an end-to-end distributed system rather than a downloadable artifact, mapping its components onto the data, optimization, parallelism, and fault-tolerance machinery of the preceding parts.
- 19.2 Scaling Laws Develops the empirical scaling laws that relate loss to parameters, data, and compute, and uses them to turn a fixed compute budget into a compute-optimal choice of model and dataset size.
- 19.3 Distributed Dataset Construction Builds the petabyte-scale pipeline that crawls, extracts, filters, and shards a corpus of trillions of tokens across a cluster, because no single machine can hold or process the data a frontier run consumes.
- 19.4 Distributed Deduplication and Data Quality Derives MinHash near-duplicate detection with the banding trick, builds a three-stage quality-filter pipeline, provides a distributed cost model for trillion-token corpora, and situates the FineWeb and SemDeDup approaches as the 2024-2025 production recipes.
- 19.5 Tokenization at Scale Trains and applies a vocabulary across a dataset too large for any one machine, turning raw text into the token stream that feeds the run and reasoning about the throughput this preprocessing step must sustain.
- 19.6 Orchestrating Distributed Pretraining Composes data, tensor, pipeline, sharded, and expert parallelism with the checkpointing and elasticity of the previous chapter, and develops the stability engineering and loss-spike recovery that babysit a run too expensive to repeat.
- 19.7 Distributed Fine-Tuning Distributes the adaptation of a pre-trained base model across machines, placing parameter-efficient methods such as LoRA correctly as a per-node enabler within the larger distributed picture.
- 19.8 Distributed Alignment: A Systems View Takes a systems view of supervised fine-tuning, reinforcement learning from human feedback, and direct preference optimization, treating alignment as a distributed-systems problem of moving data between models rather than a pure-algorithm question.
- 19.9 Energy, Cost, and Responsible Scaling Examines DeepSeek-V3 as the canonical 2025 open frontier recipe: 671B MoE, FP8 pretraining, auxiliary-loss-free routing, and a $5.6M training run that achieves frontier quality at 3-5x lower cost than a dense equivalent.
What's Next?
This chapter trained, adapted, and aligned a foundation model by composing every axis of parallel deep learning into one supervised run, but it treated the data as a fixed corpus that arrives, gets cleaned, and is consumed. The next chapter relaxes that assumption in the most demanding way possible. Chapter 20: Distributed Reinforcement Learning Infrastructure turns to the setting where the training data does not exist until the model itself generates it, where actors roll out experience against an environment while learners consume it, and where the central engineering problem becomes keeping sampling throughput and learning throughput in balance across a cluster. The alignment-by-reinforcement-learning machinery you met at the end of this chapter is one instance of that infrastructure; Chapter 20 builds it in full, developing the actor-learner architecture, distributed experience collection and replay, off-policy correction at scale, and the synchronous-versus-asynchronous tradeoffs that decide whether an RL system saturates its accelerators or starves them. Read it next, and watch the data pipeline of this chapter become a feedback loop that the model itself drives.
Bibliography & Further Reading
Scaling Laws and Compute-Optimal Training
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D. "Scaling Laws for Neural Language Models." arXiv:2001.08361, 2020. arxiv.org/abs/2001.08361
The paper that established the power-law relationship between loss, parameters, data, and compute, the empirical foundation for sizing a run in Section 19.2.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., et al. "Training Compute-Optimal Large Language Models (Chinchilla)." arXiv:2203.15556, 2022. arxiv.org/abs/2203.15556
The Chinchilla result that corrected the parameter-to-token ratio for compute-optimal training, the direct reference for the budget-allocation reasoning of Section 19.2.
Foundation Models and Pre-Training Recipes
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al. "Language Models are Few-Shot Learners (GPT-3)." arXiv:2005.14165, 2020. arxiv.org/abs/2005.14165
The 175-billion-parameter model that demonstrated emergent few-shot ability at scale, the motivating example for the orchestration challenge of Section 19.6.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv:2307.09288, 2023. arxiv.org/abs/2307.09288
An openly documented pre-training and alignment recipe, a concrete reference for the end-to-end pipeline of Sections 19.6 through 19.8.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., et al. "The Llama 3 Herd of Models." arXiv:2407.21783, 2024. arxiv.org/abs/2407.21783
A detailed account of a frontier-scale data, training, and infrastructure stack, the closest published mirror of the whole-chapter system view.
Dataset Construction, Deduplication, and Data Quality
Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., Wolf, T. "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale." arXiv:2406.17557, 2024. arxiv.org/abs/2406.17557
A reproducible, large-scale web-data pipeline with ablations on filtering and deduplication, the closest reference for the dataset construction of Sections 19.3 and 19.4.
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., Carlini, N. "Deduplicating Training Data Makes Language Models Better." arXiv:2107.06499, 2021. arxiv.org/abs/2107.06499
The study showing that near-duplicate removal improves models and reduces memorization, the motivating result for the distributed deduplication of Section 19.4.
Fine-Tuning and Parameter-Efficient Adaptation
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685, 2021. arxiv.org/abs/2106.09685
The low-rank adapter method that makes fine-tuning cheap by training a tiny fraction of parameters, the per-node enabler placed in the distributed picture of Section 19.7.
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. "QLoRA: Efficient Finetuning of Quantized LLMs." arXiv:2305.14314, 2023. arxiv.org/abs/2305.14314
The method that fine-tunes a quantized base model with low-rank adapters to fit large models on a single accelerator, extending the per-node enabler of Section 19.7.
Alignment: SFT, RLHF, and DPO
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., et al. "Training Language Models to Follow Instructions with Human Feedback (InstructGPT)." arXiv:2203.02155, 2022. arxiv.org/abs/2203.02155
The paper that established the supervised-tuning-plus-RLHF alignment recipe, the systems backbone for the alignment view of Section 19.8.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv:2305.18290, 2023. arxiv.org/abs/2305.18290
The method that aligns a model directly from preference pairs without a separate reward model or RL loop, a simpler alternative weighed in Section 19.8.
Energy, Cost, and Responsible Scaling
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., Dean, J. "Carbon Emissions and Large Neural Network Training." arXiv:2104.10350, 2021. arxiv.org/abs/2104.10350
The study that quantifies the energy and carbon cost of large training runs and the factors that move them, the foundation for the responsible-scaling accounting of Section 19.9.