Energy, Cost, and Responsible Scaling

"They asked me to scale the model. Nobody asked who pays the power bill, who owns the data, or who answers when it goes wrong. So I learned to ask all three before I allocate a single node."

A Scheduler That Reads the Electricity Meter

The earlier sections of this chapter treated scale as a systems problem: fit the model across devices, keep the workers fed, survive the failures, stop at the compute-optimal point. Underneath every one of those decisions sits a meter that never stops running. A modern frontier pretraining run consumes tens of millions of GPU-hours, costs on the order of tens to hundreds of millions of dollars, and draws power at the megawatt scale for weeks or months. At that magnitude, cost and energy stop being something you tally afterward and become parameters you design against from the first planning meeting, exactly the cost-aware posture introduced in Chapter 3 and made measurable in the cost and energy accounting of Chapter 5. This section gives you the arithmetic to put a number on a run before you launch it, and the vocabulary to defend that number to the people who pay for it and live with it.

1. The Bill: Cost as a First-Class Constraint Beginner

Start with the simplest of the three meters, because it is the one every stakeholder already understands. The dollar cost of a training run is, to first order, the price of the compute it consumes:

where $G$ is the number of GPU-hours, $p$ is the price per GPU-hour (a rental rate, or the amortized cost of owned hardware plus power), $C_{\text{useful}}$ is the useful compute the science requires in FLOPs, $F_{\text{peak}}$ is the peak throughput of one accelerator, and MFU is the model FLOPs utilization from the efficiency discussion earlier in this chapter. The second equation is the load-bearing one: the science fixes $C_{\text{useful}}$, but the GPU-hours you actually pay for are that number divided by how much of each accelerator's peak you sustain. Doubling MFU halves the bill for the identical model. This is why the utilization fights of the previous sections were never merely about speed; every percentage point of MFU is a percentage point off an eight-figure invoice.

Treating cost as a design constraint changes decisions you might otherwise make on instinct. It argues against renting the largest accelerator when a cheaper one at higher utilization is more dollars-efficient; it argues for spot and preemptible capacity wherever the elastic, fault-tolerant training of Chapter 18 lets you survive the interruptions; and it argues, most of all, for not buying compute the model cannot turn into quality, the over-training trap we return to in Section 4.

2. The Footprint: Energy and Carbon Intermediate

The second and third meters convert the same GPU-hours into physical quantities. Energy is the cleanest to reason about. If each accelerator draws an average power $P_{\text{gpu}}$ (in kilowatts) while training, the electricity delivered to the chips is $G \cdot P_{\text{gpu}}$ kilowatt-hours. But the chips are not the whole datacenter: cooling, power conversion, and distribution all draw current too. The industry captures that overhead in a single multiplier, the power usage effectiveness (PUE), defined as total facility energy divided by IT energy. A PUE of $1.1$ means the building spends ten percent on top of the compute; older facilities run $1.5$ or worse. The facility energy is therefore

Carbon follows by multiplying energy by the carbon intensity of the electricity, $I$, measured in grams of CO$_2$-equivalent per kilowatt-hour. A coal-heavy grid sits near $700$ gCO$_2$e/kWh; a grid rich in nuclear, hydro, wind, and solar can fall below $50$. The emitted carbon is

Two of these factors, PUE and $I$, are properties of where and when you train, not of your model. That observation is the seed of carbon-aware scheduling: the same job placed in a region with a cleaner grid, or deferred to an hour when wind and solar are abundant, emits dramatically less carbon for an unchanged dollar and energy cost. We treat the joules-per-token of a run as a quantity to be engineered down, and the carbon as a quantity to be both engineered down and placed wisely. The responsible-scaling and environmental-accounting practices that formalize this carbon-aware placement are developed in Chapter 35.

from codecarbon import EmissionsTracker

tracker = EmissionsTracker(project_name="frontier-pretrain")  # auto-detects region + grid
tracker.start()
train(model, data, steps=...)          # your existing training loop, unchanged
emissions_kg = tracker.stop()          # kgCO2e, also logged to emissions.csv
print(f"this run emitted {emissions_kg:.1f} kgCO2e")

3. The Levers: One Set of Knobs Cuts All Three Intermediate

The structure of the three formulas hides a gift. Dollars, energy, and carbon all scale with the GPU-hours $G$, and $G$ shrinks with efficiency. So a single set of levers, the ones this chapter has already taught for performance, doubles as the lever set for cost and footprint. The demo below makes this concrete: it fixes the useful compute of a frontier-scale run and asks what that identical run costs in all three currencies as we raise model FLOPs utilization and switch to FP8.

def estimate(gpu_hours, gpu_power_kw, pue, grid_gco2_per_kwh, price_per_gpu_hour):
    # Energy delivered to the GPUs, then scaled up by datacenter overhead (PUE).
    it_energy_kwh = gpu_hours * gpu_power_kw          # raw GPU electricity
    facility_kwh = it_energy_kwh * pue                # + cooling, power loss
    carbon_kg = facility_kwh * grid_gco2_per_kwh / 1000.0   # gCO2e -> kgCO2e
    dollars = gpu_hours * price_per_gpu_hour          # rental / amortized cost
    return dollars, facility_kwh, carbon_kg

# Total useful FLOPs are fixed; GPU-hours = useful_FLOPs / (peak_FLOPS * MFU).
useful_pflop_days = 4.5e5        # ~450k PFLOP-days of useful compute (fixed)
peak_pflops = 0.99              # H100 BF16 dense peak, PFLOP/s per GPU
gpu_power_kw = 0.70            # ~700 W per GPU under load
pue = 1.12                    # efficient modern datacenter
grid_gco2_per_kwh = 369.0     # average grid carbon intensity (gCO2e/kWh)
price_per_gpu_hour = 2.50     # USD per GPU-hour

def gpu_hours_for(mfu, fp8_speedup):
    eff_pflops = peak_pflops * mfu * fp8_speedup     # sustained PFLOP/s per GPU
    return useful_pflop_days * 86400.0 / eff_pflops / 3600.0

scenarios = [
    ("Baseline  BF16, MFU 35%", 0.35, 1.0),
    ("Tuned     BF16, MFU 50%", 0.50, 1.0),
    ("FP8       FP8,  MFU 50%", 0.50, 1.6),  # FP8 ~1.6x effective throughput
]
for name, mfu, fp8 in scenarios:
    gh = gpu_hours_for(mfu, fp8)
    d, kwh, ckg = estimate(gh, gpu_power_kw, pue, grid_gco2_per_kwh, price_per_gpu_hour)
    print(f"{name:<26}{gh:>12,.0f}{d:>13,.0f}{kwh:>13,.0f}{ckg/1000:>10.1f}")

scenario                     GPU-hours       cost $   energy kWh     tCO2e
------------------------------------------------------------------------------------
Baseline  BF16, MFU 35%     31,168,831   77,922,078   24,436,364    9017.0
Tuned     BF16, MFU 50%     21,818,182   54,545,455   17,105,455    6311.9
FP8       FP8,  MFU 50%     13,636,364   34,090,909   10,690,909    3944.9
------------------------------------------------------------------------------------
FP8 + tuned MFU vs baseline: cost x2.29, energy x2.29, carbon x2.29 lower
Same FP8 run on a low-carbon grid (30 gCO2e/kWh): carbon x12.3 lower again

The lesson is structural, not numerical. Because all three meters are driven by the same GPU-hours, the engineer who optimizes for utilization and precision is, without any extra work, also the engineer who minimizes the carbon footprint. The two goals that are sometimes posed as a trade-off, performance versus sustainability, are the same goal viewed through different units. Higher MFU (the utilization arithmetic of Section 19.6), FP8 and low precision (Chapter 15), compute-optimal sizing, and not over-training all push the denominator up, and the denominator is shared.

4. Compute-Optimal and the Cost of Over-Training Intermediate

The fourth lever is the subtlest and the one most often gotten wrong: spending compute the model cannot convert into quality. The compute-optimal framing developed earlier in this chapter answers a precise question, given a fixed compute budget $C$, how should you split it between model size $N$ and training tokens $D$, subject to $C \approx 6 N D$, to minimize loss. Train far past that point and the loss curve flattens while the meters keep climbing. Every additional token then buys diminishing quality at undiminished cost, energy, and carbon.

There is a legitimate reason to over-train relative to the pretraining-optimal point: a model that will serve billions of inference requests should be made smaller and trained longer, because a smaller model is cheaper to serve forever, and the extra training cost is amortized over the deployment lifetime. That is an inference-economics decision, made deliberately with the serving cost of Chapter 3 in view, not an accident. The failure mode this section warns against is the accidental over-train: running longer because the cluster was already booked, or because a flat loss curve was mistaken for one still descending. Naming the compute-optimal point, and justifying any departure from it in writing, is how a cost-aware team keeps a training plan honest.

5. Responsible Scaling: Provenance, Safety, and Governance Advanced

Cost and energy bound a run from below; responsibility bounds it from the sides. Three duties attach to operating at this scale, and a serious training plan treats them as gates, not afterthoughts. The first is data provenance and consent. The corpus construction, deduplication, and tokenization of the early sections of this chapter determine not only quality but also legitimacy: where the data came from, whether its use is licensed or consented, and whether personal or copyrighted material was filtered. A model is downstream of its data in every sense, including the legal and ethical ones, so provenance must be recorded as the corpus is built, not reconstructed after a dispute.

The second duty is safety evaluation before release. A frontier model is evaluated not only for capability but for the harms it might enable, and that evaluation happens before weights are shipped, using the held-out, contamination-controlled methodology of Chapter 5. The third is compute governance: because a frontier run requires a concentration of accelerators that few organizations control, the training compute itself has become a unit of policy, with disclosure thresholds and review tied to total FLOPs. An engineer planning a run at this scale should know which thresholds it crosses and what reporting they trigger.

6. Chapter 19, Composed Beginner

This chapter set out to train a foundation model at scale, and the path it walked was, in effect, the whole book composed into a single run. The data was constructed, deduplicated, and tokenized using the distributed data processing of Part II. The training distributed data, the model, and the optimizer state across thousands of accelerators using the 3D and 4D parallelism of Chapters 15 through 17, and it survived weeks of inevitable hardware failure using the elastic, fault-tolerant machinery of Chapter 18. The run was sized to the compute-optimal point, then fine-tuned and aligned into something usable, and at every step it was bounded by the cost and energy meters of this final section. A foundation model is not one technique; it is the entire discipline of scale-out AI, exercised at once and paid for in three currencies.

DeepSeek-V3: The 2025 Open Frontier Recipe Advanced

The cost-accounting arithmetic above becomes tangible when applied to a specific public run. DeepSeek-V3, released in January 2025, is the canonical 2025 open frontier recipe: the first publicly documented industrial training run to combine FP8 pretraining, mixture-of-experts (MoE) routing, and full 4D parallelism at frontier scale, with a cost that surprised the field.

The model has 671 billion total parameters but activates only 37 billion per token through top-2 expert routing: each forward pass touches 2 of 256 routed experts plus a shared expert layer, so the compute per token tracks a dense 37B model while the capacity tracks a much larger one. The run trained on 14.8 trillion tokens across 2,048 NVIDIA H800 GPUs for a reported 2.788 million GPU-hours, at a total estimated cost of approximately $5.6 million at prevailing cloud rates. That figure is roughly an order of magnitude below contemporaneous dense-model runs of comparable benchmark quality, and it sits at the low end of the cost range this section's arithmetic would predict for 37B active parameters at high utilization.

Four engineering choices separate DeepSeek-V3 from earlier frontier recipes, each connecting directly to material in this book.

First, FP8 pretraining at scale. DeepSeek-V3 is the first publicly documented industrial run to train the full forward and backward pass in FP8 and achieve BF16-equivalent final quality, validating the low-precision path described in section 15.8 for a production frontier model rather than a proof-of-concept. The key was fine-grained tile-level quantization and delayed rescaling, which stabilized the training signal across the full 14.8 trillion token run without any quality recovery phase at BF16. The throughput gain corresponds to roughly 1.5 to 1.8 times the effective FLOP/s relative to BF16 on the same hardware, which the cost figures above imply directly.

Second, multi-head latent attention (MLA). DeepSeek-V3 replaces standard multi-head attention with a low-rank projection that compresses the key-value cache by approximately 93 percent relative to multi-head attention at equivalent model dimension. For inference, that compression is transformative: the KV cache that would otherwise consume tens of gigabytes per concurrent sequence shrinks to a few gigabytes, enabling significantly higher batch sizes. For training, the same projection reduces activation memory during the backward pass, freeing headroom for larger microbatches within the same device memory budget.

Third, auxiliary-loss-free MoE routing. Standard MoE load balancing uses an auxiliary loss term added to the training objective to discourage all tokens routing to a small set of experts. DeepSeek-V3 replaces this with a bias-update method: each expert carries a learned scalar bias that is updated outside the gradient path to keep token counts balanced across experts, leaving the primary training objective uncontaminated. The approach was introduced in Chapter 17's treatment of MoE load balancing and is developed further in section 17.6. The practical result is that expert utilization stays balanced throughout the run without the gradient interference that auxiliary losses introduce at scale.

Fourth, full 4D parallelism. The run composes data parallelism, pipeline parallelism, expert parallelism, and tensor parallelism simultaneously across the 2,048 H800 cluster. This is the same four-dimensional composition described in section 19.6 for frontier orchestration, with expert parallelism layered on top of the 3D baseline described in Chapters 15 and 16. The FSDP2 and ZeRO-3 sharding of optimizer state (section 16.5) ensures the 671B parameter count fits within the aggregate device memory of the cluster without replication.

The following practical-example callout quantifies the cost advantage relative to a hypothetical equivalent run without these two key choices.

The research-frontier callout below points to the primary citations and the follow-on systems work that extends DeepSeek-V3's FP8 MoE approach to next-generation hardware.