"I am the deploy that touched four hundred replicas and lived. Some of them got the new weights first, some got them last, and for ninety seconds the fleet disagreed with itself about what model it was. The rollback plan was the only reason I slept that night."
A Release That Has Made Peace With Partial Rollout
Chapter Overview
This chapter takes everything Part V has built, the inference engines, the serving fleet, the retrieval pipeline, and asks how the whole apparatus is operated in production over time. A distributed AI system is not deployed once; it is deployed continuously, watched continuously, and changed continuously, and every one of those activities is itself a distributed system. The nine sections develop MLOps and its language-model variant LLMOps as a set of practices that must scale out across a fleet: versioning and rolling out models, making training and data pipelines reproducible, tracking experiments from many workers, monitoring and observing the fleet, detecting drift on live traffic, validating new versions safely, and recovering from failure. Where earlier chapters asked how to serve a model at scale, this chapter asks how to keep the entire serving organization healthy as it evolves.
Read in order, the nine sections take you from "MLOps is a dashboard and a deploy button" to a working mental model of the production loop as a distributed system: operate a fleet of training, serving, and retrieval machines as one managed organism, version the data and training pipelines so any model can be reproduced, register models and prompts as named and versioned objects, push changes through automated continuous integration and delivery, collect experiment metrics from thousands of distributed runs, observe the whole fleet through aggregated telemetry, detect distribution drift on live traffic, roll out new versions behind A/B tests and shadow traffic, and recover from regressions with coordinated rollbacks and guardrails. The argument is cumulative and it closes Part V's arc: the serving fleet of Chapter 24 and the retrieval pipeline of Chapter 25 become the production workloads that this operational loop must keep running, and the streaming and data-versioning patterns of Part II reappear as the pipelines and telemetry that feed it.
Prerequisites
This chapter assumes the serving chapters of Part V and the data foundations of Part II. From Chapter 23: Distributed Inference Systems through the LLM serving and retrieval chapters that follow it you carry the workloads this chapter operates: the inference fleet, the serving replicas, and the retrieval pipeline are exactly the production systems that MLOps must deploy, monitor, and recover. From Chapter 8: Distributed Storage and Data Loading you carry the reproducibility and data-versioning foundations that make a training pipeline repeatable: how datasets are stored, versioned, and loaded across machines, which becomes the substrate for the reproducible pipelines of Section 26.2 and the registries of Section 26.3. The chapter also leans on the stream-processing patterns of Part II, since fleet-wide telemetry and drift detection are streaming aggregations over live traffic, and on the evaluation vocabulary of Part I, since A/B testing and shadow deployment are evaluation methods scaled to production. Beyond these the chapter assumes comfortable Python, a working picture of how a model is trained and served, and the distributed-systems vocabulary of replication, coordination, and consistency that the whole book has been building. No prior experience with a specific MLOps platform is needed; Section 26.1 builds the why-operations-are-distributed argument from the ground up before any tool appears.
Learning Objectives
- Explain why operating an AI system across a fleet is the core MLOps problem, in particular how replication, coordination, and continuous change turn every operational task into a distributed-systems problem.
- Make a distributed data and training pipeline reproducible, versioning data, code, and configuration so that any deployed model can be rebuilt from a known state.
- Design model and prompt registries that name, version, and stage the artifacts a fleet runs, and reason about how a registry serves consistent versions to many replicas.
- Build a CI/CD pipeline for distributed ML that tests, validates, and promotes models and serving code through automated continuous integration and delivery.
- Track experiments across thousands of distributed runs, aggregating metrics, parameters, and artifacts from many workers into one queryable store.
- Construct fleet-wide monitoring and observability that aggregates telemetry from every node and surfaces regressions before users notice them.
- Detect distribution drift across shards of live traffic, treating drift detection as a streaming aggregation rather than a one-off batch computation.
- Validate new model versions safely with A/B testing and shadow deployment at scale, reasoning about traffic splitting, statistical power, and risk.
- Recover from failure with coordinated rollbacks, incident response, and guardrails that keep a distributed AI system safe as it changes in production.
Chapter Roadmap
- 26.1 Operating AI Across a Fleet Frames MLOps as the discipline of operating an AI system spread across many machines, showing how replication, coordination, and continuous change turn every operational task into a distributed-systems problem.
- 26.2 Distributed Data and Training Pipelines Makes the data and training pipelines that produce models reproducible across many machines, versioning data, code, and configuration so any deployed model can be rebuilt from a known state.
- 26.3 Model and Prompt Registries Builds the registries that name, version, and stage the models and prompts a fleet runs, reasoning about how a registry serves consistent versions to hundreds of replicas.
- 26.4 CI/CD for Distributed ML Develops continuous integration and delivery for distributed machine learning, automating the tests, validations, and promotions that move a model and its serving code into production.
- 26.5 Distributed Experiment Tracking Tracks experiments across thousands of distributed runs, aggregating metrics, parameters, and artifacts from many workers into one queryable store that makes results comparable and reproducible.
- 26.6 Fleet-Wide Monitoring and Observability Constructs observability that aggregates telemetry from every node into fleet-wide views, surfacing latency, throughput, and quality regressions before users notice them.
- 26.7 Distributed Drift Detection Defines data drift and concept drift formally, derives KS, PSI, and MMD statistical tests with simulation code, builds a false-positive cost model, and computes the PSI threshold that minimises total 30-day cost given retrain cost and accuracy-decay rate.
- 26.8 A/B Testing and Shadow Deployment at Scale Validates new model versions safely by splitting live traffic for A/B tests and mirroring it to shadow deployments, reasoning about traffic routing, statistical power, and risk.
- 26.9 Rollbacks, Incident Response, and Guardrails Closes the loop with coordinated rollbacks, incident response, and runtime guardrails that keep a distributed AI system safe and recoverable as it changes in production.
What's Next?
This chapter closed the production loop around everything Part V built, the operational discipline that versions, deploys, monitors, and recovers a distributed AI system as it changes in production, and with it Part V comes to an end: the per-node inference engine, the distributed inference system, the serving fleet, the retrieval pipeline, and now the MLOps loop that keeps them all running. The book now turns from systems that serve a single model to systems made of many cooperating intelligences. Chapter 27: Distributed Artificial Intelligence opens Part VI by stepping up a level of abstraction: instead of distributing the computation of one model across machines, it distributes the intelligence itself across many autonomous agents that perceive, decide, and act, then coordinate to solve problems no single agent could. Where Part V asked how to operate one distributed model in production, Part VI asks what happens when the distributed pieces are themselves decision-makers. The serving and operational machinery developed across this part becomes the infrastructure on which those agents run. Read it next, and watch the unit of distribution change from a tensor to an agent.
Bibliography & Further Reading
Foundations of Production ML and MLOps
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., Dennison, D. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems (NeurIPS), 2015. papers.nips.cc
The paper that named the maintenance cost of production machine-learning systems and the systems-level traps behind it, the foundational motivation for the operating-a-fleet framing of Section 26.1.
Breck, E., Cai, S., Nielsen, E., Salib, M., Sculley, D. "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction." IEEE International Conference on Big Data, 2017. research.google
A practical rubric of tests for data, models, infrastructure, and monitoring that a production ML system should pass, the reference for the CI/CD and validation practices of Section 26.4.
Polyzotis, N., Roy, S., Whang, S. E., Zinkevich, M. "Data Management Challenges in Production Machine Learning." Proceedings of the 2017 ACM SIGMOD International Conference on Management of Data, 2017. dl.acm.org/doi/10.1145/3035918.3054782
A survey of the data-management problems that production ML faces, validation, versioning, and drift among them, the reference for the reproducible pipelines of Section 26.2 and the drift detection of Section 26.7.
Paleyes, A., Urma, R.-G., Lawrence, N. D. "Challenges in Deploying Machine Learning: A Survey of Case Studies." arXiv:2011.09926, 2020 (ACM Computing Surveys 2022). arxiv.org/abs/2011.09926
A survey of real-world deployment case studies and the operational failures behind them, the reference for the incident-response and end-to-end lifecycle concerns of Sections 26.1 and 26.9.
Pipelines, Tracking, and Platforms
Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., Xie, F., Zumar, C. "Accelerating the Machine Learning Lifecycle with MLflow." IEEE Data Engineering Bulletin, 2018. mlflow.org
The system paper for MLflow, the open-source platform for experiment tracking, model registry, and lifecycle management, the primary reference for the tracking of Section 26.5 and the registries of Section 26.3.
Google Cloud. "MLOps: Continuous Delivery and Automation Pipelines in Machine Learning." Google Cloud Architecture Center. cloud.google.com/architecture
The reference architecture that defines MLOps maturity levels and the continuous-delivery pipeline for ML, the framing reference for the CI/CD material of Section 26.4.
Kubeflow Authors. "Kubeflow Documentation." The Kubeflow Project. kubeflow.org/docs
The documentation for the Kubernetes-native ML platform that runs distributed training and serving pipelines, a working reference for the fleet operations and pipelines of Sections 26.1 and 26.2.
Monitoring, Drift, and Observability
Evidently AI. "Evidently AI Documentation." Evidently AI. docs.evidentlyai.com
The documentation for an open-source library that computes data drift, target drift, and model-quality reports over production data, a hands-on reference for the drift detection of Section 26.7 and the monitoring of Section 26.6.
OpenTelemetry Authors. "Semantic Conventions for Generative AI Systems." OpenTelemetry. opentelemetry.io
The emerging standard for tracing and metrics conventions specific to generative-AI systems, the reference for the fleet-wide telemetry and observability of Section 26.6.
Guardrails and Safety in Production
Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., Khabsa, M. "Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations." arXiv:2312.06674, 2023. arxiv.org/abs/2312.06674
The model that classifies prompts and responses against a safety taxonomy as a deployable input-output filter, a core reference for the runtime guardrails of Section 26.9.
NVIDIA. "NeMo Guardrails." GitHub. github.com/NVIDIA/NeMo-Guardrails
An open-source toolkit for adding programmable safety and topic guardrails around LLM applications, a practical reference for the production guardrails of Section 26.9.