Quantization

"They asked me to hold sixteen bits per weight, then four. I lost twelve bits and almost none of my opinions."

A Weight Matrix, Newly Compressed

The previous section established the rule that governs this whole chapter: the efficiency of one node sets the unit cost that the fleet multiplies, so a per-node saving is never just a local optimization. Quantization is the first and largest of those savings. It does not change what the model computes in any deep sense; it changes how precisely each number is stored, and modern large models tolerate a surprising amount of imprecision in their weights with negligible loss of quality. The result is the single most cost-effective lever in inference serving, which is why every production stack, from a phone to a datacenter, reaches for it before anything else.

To see why fewer bits buy speed and not just space, recall the roofline model of Chapter 3. Autoregressive generation produces one token at a time against a batch that is often small, so each weight is read from memory and used for only a handful of arithmetic operations before the next weight is needed. The accelerator's arithmetic units sit idle waiting for bytes; the workload lives far to the left of the roofline's ridge, firmly memory-bound. Halving the bytes per weight roughly halves the time spent waiting, which is why quantization speeds up generation even though it adds a tiny amount of arithmetic to rescale the integers back to a usable range.

1. The Quantization Map: Floats Into Integers and Back Intermediate

Quantization is an affine map between a continuous range of real values and a small set of integers. Pick a range of weight values to represent, choose a number of bits $b$, and divide the range into $2^b$ evenly spaced levels. A floating-point weight $w$ becomes an integer $q$, and the integer can be turned back into an approximate float $\hat{w}$, through a scale $s$ and an optional zero-point $z$:

The scale $s$ is the width of one quantization step in the original units; the zero-point $z$ is the integer that represents a true zero, needed when the range is not symmetric about the origin. For weights, which are usually centered near zero, a symmetric scheme with $z = 0$ and a signed integer range is common: an 8-bit signed integer covers $[-128, 127]$, so $q_{\max} = 127$ and the scale is $s = \max|w| / 127$ over whatever group of weights shares that scale. The reconstruction error $w - \hat{w}$ is bounded by half a step, $s/2$, which makes the message of the formula plain: the error scales with $s$, and $s$ is set by the largest magnitude in the group divided by the integer range. One large outlier inflates $s$ and coarsens the quantization of every other weight that shares it. Figure 22.2.1 shows the mapping and the memory it saves.

2. Per-Tensor Versus Per-Channel, and the Outlier Problem Intermediate

The choice of which weights share a scale is where most of the accuracy is won or lost. The cheapest option is per-tensor quantization: one scale for the entire weight matrix. It is compact, one extra number per matrix, but it is fragile, because a single large entry anywhere in the matrix sets $s$ for all of it. The finer option is per-channel (or per-group) quantization: a separate scale for each row, column, or small block of the matrix. It costs a few hundred extra scale values, a rounding error in the total memory, and it isolates a large entry so that it only coarsens its own channel instead of the whole tensor. The numerical formats that make these integer and low-bit-float representations possible are the subject of Chapter 15; here we use them to shrink a deployed model.

Outliers are not a rare nuisance in large transformers; they are systematic. Certain feature dimensions in the activations of large language models carry values tens of times larger than their neighbors, and these outlier channels dominate any shared scale. The activation outlier problem is the central difficulty in pushing quantization below 8 bits, and it has produced a family of methods that either give outlier-bearing channels their own finer scale (per-channel and per-group schemes) or move the difficulty from activations to weights. SmoothQuant, for instance, rescales activations and weights jointly so that the hard-to-quantize outliers are absorbed into the weights, where per-channel scales handle them, leaving the activations smooth enough for plain 8-bit integers. The code below makes the per-tensor versus per-channel gap concrete on a matrix with deliberately injected outliers.

import numpy as np

rng = np.random.default_rng(0)
rows, cols = 512, 512                                  # a weight matrix W: out x in
W = rng.standard_normal((rows, cols)).astype(np.float32)
# Inject heavy-tailed outliers into a few columns (an input-channel effect).
outlier_cols = rng.choice(cols, size=8, replace=False)
W[:, outlier_cols] *= 20.0


def quant_dequant(W, bits, axis):
    qmax = 2 ** (bits - 1) - 1                          # symmetric signed range
    if axis is None:                                    # per-tensor: one scale
        scale = np.max(np.abs(W)) / qmax
        q = np.clip(np.round(W / scale), -qmax - 1, qmax)
        return q * scale, np.array([scale])
    amax = np.max(np.abs(W), axis=axis, keepdims=True)  # per-channel: one scale/row
    scale = amax / qmax
    q = np.clip(np.round(W / scale), -qmax - 1, qmax)
    return q * scale, scale.ravel()


def report(name, bits, axis):
    Wq, scales = quant_dequant(W, bits, axis)
    err = np.linalg.norm(W - Wq) / np.linalg.norm(W)    # relative reconstruction error
    fp16_bytes = W.size * 2
    int_bytes = W.size * bits / 8 + scales.size * 4     # packed ints + fp32 scales
    print(f"{name:<26} rel_err={err:.4f}  "
          f"mem={int_bytes/1024:7.1f} KiB  "
          f"shrink_vs_fp16={fp16_bytes/int_bytes:4.1f}x")


print(f"fp16 baseline              mem={W.size*2/1024:7.1f} KiB")
report("INT8 per-tensor",  8, None)
report("INT8 per-channel", 8, 0)
report("INT4 per-tensor",  4, None)
report("INT4 per-channel", 4, 0)

fp16 baseline              mem=  512.0 KiB
INT8 per-tensor            rel_err=0.0702  mem=  256.0 KiB  shrink_vs_fp16= 2.0x
INT8 per-channel           rel_err=0.0079  mem=  258.0 KiB  shrink_vs_fp16= 2.0x
INT4 per-tensor            rel_err=0.4021  mem=  128.0 KiB  shrink_vs_fp16= 4.0x
INT4 per-channel           rel_err=0.1450  mem=  130.0 KiB  shrink_vs_fp16= 3.9x

Two lessons sit in Output 22.2.1. First, the memory reduction is exactly what the bit count promises: INT8 halves it, INT4 quarters it, and the per-channel scales add a negligible two KiB. Second, the accuracy is governed entirely by how the scale is shared. Per-tensor INT4 destroys the matrix (a $40\%$ relative error) because eight outlier columns set a scale that crushes the other five hundred; per-channel INT4 confines the damage and recovers most of the fidelity at the same compression. This is the empirical core of the outlier story: the bits are cheap, the scale assignment is everything.

3. PTQ, QAT, and the LLM-Specific Methods Advanced

There are two ways to obtain a quantized model. Post-training quantization (PTQ) takes a finished model and quantizes it directly, using a small calibration set, a few hundred unlabeled examples, to observe the typical range of activations and set the scales. PTQ is fast, needs no labels and no gradient, and is the default for large models because retraining them is expensive. Quantization-aware training (QAT) instead simulates the rounding during training or fine-tuning, inserting fake-quantize operations into the forward pass so the optimizer learns weights that are robust to the eventual precision loss. QAT recovers more accuracy, especially at very low bit widths, but it costs a training run, so it is reserved for cases where PTQ's accuracy is not good enough and the model is small enough to retrain.

For large language models specifically, weight-only PTQ has converged on a few high-quality methods. GPTQ quantizes the weights one column at a time and, after rounding each column, adjusts the not-yet-quantized columns to compensate for the error just introduced, an error-correcting sweep based on second-order (Hessian) information that keeps the layer's output close to the original. AWQ (activation-aware weight quantization) starts from the observation that a small fraction of weight channels, the ones multiplied by large activations, matter far more than the rest; it scales those salient channels up before quantizing so they survive the rounding, protecting accuracy without storing them in higher precision. Both are weight-only and post-training, and both target the same regime: 4-bit weights with 16-bit activations, the standard for memory-bound LLM serving.

Why weight-only INT4 specifically? Because in autoregressive generation the weights are read from memory every single token while the activations of a single small batch are tiny by comparison. The weights dominate both the storage and the traffic, so quantizing them to 4 bits captures almost the entire benefit, and leaving activations at 16 bits sidesteps the worst of the activation-outlier problem. The accuracy-versus-compression trade-off is therefore favorable in exactly this corner: 4-bit weight-only quantization typically costs a fraction of a perplexity point on a well-tuned method while quartering the model's footprint, whereas pushing weights to 3 or 2 bits, or quantizing activations too, starts to cost real quality and needs the heavier machinery of QAT or careful per-group outlier handling.

4. Why This Matters for Serving: From One Node to a Fleet Beginner

Everything above is a single-node, scale-up technique, and that is precisely why it belongs at the front of a book about scale-out. The fleet you deploy is a multiple of one node's behavior, so a saving on the node is a saving on the fleet, multiplied. A 4-bit model needs a quarter of the weight memory, so it fits on a quarter (or a half, accounting for the KV cache and batching room) of the accelerators per replica; with each replica cheaper, the same request volume is served by a smaller fleet at lower cost. The roofline argument adds a second win: because generation was memory-bound, the lighter traffic also raises per-node throughput, so each replica serves more, shrinking the replica count further.

This is the bridge from per-node economics to fleet sizing that the rest of Part V builds on. Chapter 23 turns these per-node numbers into a replica count and a load-balancing strategy, and Chapter 24 carries the quantized model into a distributed LLM serving stack where the saving compounds with paged KV-cache management and continuous batching. The discipline is always the same one this chapter opened with: measure the unit, then multiply. Quantization is the largest single adjustment you can make to that unit before any distribution begins.

# pip install transformers bitsandbytes
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

cfg = BitsAndBytesConfig(load_in_4bit=True,          # NF4 weight-only quantization
                         bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype="bfloat16")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B",
                                             quantization_config=cfg,
                                             device_map="auto")   # ~4x less VRAM