Scaling Laws

"They asked how big I should be. I asked how much compute they had. We were, it turned out, asking the same question."

A Model Sizing Itself to a Budget

The previous section framed a foundation-model run as a distributed system with many moving parts. This section answers the question that has to be settled before any of those parts are provisioned: given a fixed amount of compute, how large should the model be and on how many tokens should it train? For most of the field's history this was guesswork, and the guesses were biased toward large models trained for too few steps. Scaling laws replaced the guesswork with measurement. They are not a theory of why neural networks generalize; they are a robust empirical regularity, observed across many orders of magnitude, that the test loss of a transformer follows a power law in the resources you spend on it. That regularity is what makes a multi-million-dollar training run something you can plan rather than gamble on.

1. Loss as a Power Law in Parameters, Data, and Compute Intermediate

The central empirical finding, established by Kaplan and co-authors in 2020 and refined since, is that the test loss of an autoregressive transformer is, to a good approximation, a power law in each of three resources held in surplus: the number of non-embedding parameters $N$, the number of training tokens $D$, and the total training compute $C$. Holding the other resources non-binding, each relationship has the same shape,

A power law is a straight line on log-log axes, which is exactly what Figure 19.2.1 shows, and that straight-line behavior is what makes extrapolation trustworthy: if five small runs spanning two orders of magnitude in compute lie on a line, the line keeps going, and the loss of a run a hundred times larger is read off the fit rather than discovered by spending the compute. The exponents $\alpha$ are small, typically around $0.05$ to $0.10$ for parameters and data, which means the loss improves slowly; a tenfold increase in compute buys a modest absolute drop in loss, and that diminishing return is the economic reality every scaling decision lives inside. A more honest functional form adds an irreducible floor, the entropy of the data itself, that no amount of compute removes,

where $E$ is the loss a perfect model would still incur. This is the form Hoffmann and co-authors fit in the 2022 Chinchilla study, and it is the one that yields a compute-optimal recipe once we connect $N$ and $D$ through a compute budget.

2. The Compute Identity: Why $C \approx 6ND$ Intermediate

To turn the loss surface into a budget plan we need to relate the three resources, and one identity does it. The compute to train a dense transformer is, to excellent approximation,

where $C$ is total floating-point operations, $N$ is the parameter count, and $D$ is the number of training tokens. The factor of six is not a fudge: each parameter participates in roughly two FLOPs (a multiply and an add) per token in the forward pass, and the backward pass costs about twice the forward pass, giving roughly $2 + 4 = 6$ FLOPs per parameter per token. Multiplying by $N$ parameters and $D$ tokens gives $C \approx 6ND$. The rule ignores attention's quadratic term, which is small relative to the feed-forward cost at the sequence lengths used in practice, and it is accurate enough that every capacity-planning spreadsheet in the field is built on it.

This identity is the lever for distribution. Once you know your cluster's sustained throughput in FLOPs (the number of GPUs times their realized FLOPs per second times the wall-clock you can hold them), you know $C$. Then $C \approx 6ND$ is one equation in two unknowns, $N$ and $D$, and the scaling law supplies the second equation by telling you which $(N, D)$ pair on the budget curve minimizes the loss. The answer dictates the parallelism strategy directly: a large $N$ pushes you toward model and pipeline parallelism (Chapter 16) because the model no longer fits on one device, while a large $D$ pushes you toward higher data-parallel throughput (Chapter 15) so the tokens stream through fast enough to finish on time.

3. Compute-Optimal Training: The Chinchilla Correction Advanced

For a fixed compute budget $C$, substitute $D = C / (6N)$ into the loss form $L(N, D) = E + A N^{-\alpha} + B D^{-\beta}$ and minimize over $N$. The minimization yields an optimal model size and token count that both grow as power laws in the budget, $N_{\text{opt}} \propto C^{a}$ and $D_{\text{opt}} \propto C^{b}$ with $a$ and $b$ each near one half. The headline empirical result of the Chinchilla study is that, with the fitted exponents, model size and data should scale in roughly equal proportion, which corresponds to a near-constant token-to-parameter ratio of about twenty tokens per parameter at the compute-optimal point,

This corrected a costly bias. The earlier 2020 analysis had recommended spending most of a larger budget on more parameters and comparatively few extra tokens, which produced large, badly under-trained models. Chinchilla showed that a 70-billion-parameter model trained on 1.4 trillion tokens beat the 175-billion and 280-billion-parameter models of its day, because those larger models had been starved of data. The lesson for a planner is blunt: at a fixed budget, a smaller model trained on more tokens often wins, and the twenty-tokens-per-parameter heuristic is the first sanity check on any proposed run. The code below fits a power-law loss curve to a few pilot points, extrapolates to a hundredfold-larger budget, and then applies the $6ND$ rule with the twenty-to-one ratio to size a model for several budgets.

import numpy as np

# A few (compute C in FLOPs, measured test loss) points from small pilot runs.
# Loss is modeled as L(C) = E + A * C**(-alpha): an irreducible floor E plus a
# power-law term that shrinks as compute grows.
C = np.array([1e17, 3e17, 1e18, 3e18, 1e19])          # training compute (FLOPs)
L = np.array([3.21, 2.86, 2.55, 2.31, 2.12])          # observed test loss (nats)

# Fit E, A, alpha: grid over the floor E, then a linear fit in log space for
# (A, alpha), since log(L - E) = log(A) - alpha * log(C) is linear.
best = None
for E in np.linspace(0.0, min(L) - 0.01, 400):
    y, x = np.log(L - E), np.log(C)
    A_mat = np.vstack([np.ones_like(x), -x]).T
    (logA, alpha), *_ = np.linalg.lstsq(A_mat, y, rcond=None)
    sse = np.sum((E + np.exp(logA) * C ** (-alpha) - L) ** 2)
    if best is None or sse < best[0]:
        best = (sse, E, np.exp(logA), alpha)

sse, E, A, alpha = best
print("fitted power law  : L(C) = %.3f + %.3f * C^(-%.4f)" % (E, A, alpha))
print("fit SSE           : %.2e" % sse)

C_target = 1e21                                        # 100x the biggest pilot run
print("extrapolated loss : L(%.0e FLOPs) = %.3f"
      % (C_target, E + A * C_target ** (-alpha)))

# Chinchilla split: C = 6 * N * D with D = 20 * N gives N = sqrt(C / (6*20)).
def chinchilla_split(C_budget, ratio=20.0):
    N = np.sqrt(C_budget / (6.0 * ratio))
    return N, ratio * N

for C_budget in [1e21, 1e22, 1e23, 1e24]:
    N, D = chinchilla_split(C_budget)
    print("budget %.0e FLOPs -> N=%.2e params, D=%.2e tokens, 6ND=%.0e"
          % (C_budget, N, D, 6.0 * N * D))

fitted power law  : L(C) = 1.333 + 3152.716 * C^(-0.1897)
fit SSE           : 9.42e-05
extrapolated loss : L(1e+21 FLOPs) = 1.660
budget 1e+21 FLOPs -> N=2.89e+09 params, D=5.77e+10 tokens, 6ND=1e+21
budget 1e+22 FLOPs -> N=9.13e+09 params, D=1.83e+11 tokens, 6ND=1e+22
budget 1e+23 FLOPs -> N=2.89e+10 params, D=5.77e+11 tokens, 6ND=1e+23
budget 1e+24 FLOPs -> N=9.13e+10 params, D=1.83e+12 tokens, 6ND=1e+24

The fitted exponent here, near $0.19$, is steeper than the small exponents seen on real corpora because the synthetic pilot points were chosen to make the straight line easy to read; on real data the curve is flatter and the planning runs must span more compute to pin it down. The mechanics are identical, and the table is the deliverable a planning meeting actually wants: a defensible model size for each budget the team might be granted.

4. The Inference-Aware Correction: Train Smaller, Serve Cheaper Advanced

Chinchilla minimizes training loss for a fixed training budget, but a model that will be served to millions of users incurs a second, often larger, cost: inference. Every query runs the forward pass, and the forward cost scales with $N$, so a smaller model is cheaper to serve forever. This reframes the optimization. If you will serve the model heavily, it pays to push past the twenty-to-one ratio and train a smaller model on far more tokens than compute-optimality alone would suggest, accepting a higher training cost in exchange for a permanently lower inference cost. This is the logic behind the LLaMA family and many production models, which are deliberately "over-trained" relative to Chinchilla: a 7-billion-parameter model trained on one to two trillion tokens sits well past the compute-optimal ratio, but it is small enough to serve cheaply at scale, and the serving savings dwarf the extra training compute over the model's deployed life.

Formally, the right objective minimizes total lifetime compute, training plus inference, $C_{\text{train}} + C_{\text{infer}}$, where $C_{\text{infer}} \approx 2 N D_{\text{infer}}$ and $D_{\text{infer}}$ is the total tokens the model will ever generate in service. When $D_{\text{infer}}$ is large, the optimum shifts toward smaller $N$, and the shift can be dramatic for a model with billions of expected queries. The connection to distribution is direct: a smaller served model needs less model parallelism per replica and fits more replicas per node, which is exactly the per-replica economics that Chapter 24 multiplies across a serving fleet, building on the per-node cost model of Chapter 22.

import numpy as np
from scipy.optimize import curve_fit

def loss_law(C, E, A, alpha):                 # L(C) = E + A * C**(-alpha)
    return E + A * np.power(C, -alpha)

C = np.array([1e17, 3e17, 1e18, 3e18, 1e19])
L = np.array([3.21, 2.86, 2.55, 2.31, 2.12])

# curve_fit jointly estimates E, A, alpha; log-domain fitting and Huber loss are
# one keyword away for robustness to noisy large runs.
(E, A, alpha), _ = curve_fit(loss_law, C, L, p0=[1.0, 1e3, 0.1], maxfev=10000)
print(f"E={E:.3f}  A={A:.1f}  alpha={alpha:.4f}")

5. Using Scaling Laws to Plan a Distributed Run Intermediate

Putting the pieces together gives a planning recipe that runs before any production training starts. Estimate the cluster's sustained throughput in FLOPs per second, multiply by the wall-clock you can hold the allocation, and subtract realistic overheads to get the budget $C$. Run a handful of pilot jobs spanning two orders of magnitude in compute, fit the loss law, and read off the predicted loss at $C$ to know whether the result will be worth the spend at all. Apply $C \approx 6ND$ with the compute-optimal ratio, adjusted downward in $N$ if the model will be served heavily, to fix the target $(N, D)$. Only then do you choose the parallelism mesh: the chosen $N$ sets how much model and pipeline parallelism you need to fit the model (Chapter 16), and the chosen $D$ sets the data-parallel width needed to stream the tokens in time (Chapter 15), with the communication-cost models of Chapter 3 telling you when adding workers stops helping.

None of this guarantees the production run succeeds; scaling laws describe the loss you reach under stable, well-tuned training, and the chapters ahead are about making that training stable and well-tuned across thousands of machines. What the laws do guarantee is that you enter the run with a defensible target rather than a hopeful guess, which is the difference between planning a foundation-model run and gambling on one. The next section turns to the first thing the chosen budget demands: assembling and preparing the trillions of tokens of $D$, a distributed-data problem in its own right, taken up in Section 19.3.