On-Device Inference

"They quantized me to four bits and dropped me into 4 GB of phone RAM. I lost a little precision and gained a billion homes."

A Quantized Model, Living in 4 GB of Phone RAM

Every form of distribution in this book so far has moved work between machines an operator owns: shards across a cluster, replicas across a serving fleet, agents across a network. On-device inference moves the model to the one machine the operator does not own, the user's own hardware, and then runs the whole computation there. The motivation is the same triad that pushed work off a single machine in Section 1.1, read in reverse: when the endpoints number in the billions, sending every query to a datacenter is the ceiling. Latency crosses a wireless link twice, privacy-sensitive data leaves the device, the service pays for every inference, and a dropped connection means no answer at all. Running the model locally removes all four problems at once, in exchange for living inside an envelope far smaller than any node in Chapter 24 ever sees.

1. Four Constraints That Have No Datacenter Remedy Beginner

A datacenter node answers resource pressure by adding more of the resource: more memory, more accelerators, more power and cooling. The device cannot. Its constraints are fixed at manufacture and shared with everything else the user is doing, so the model must be shaped to fit them rather than the other way around. Four of them bind, and they bind together.

Memory. A phone may report 8 GB of RAM, but the operating system, the foreground app, and the graphics surface claim most of it; an inference engine that demands more than a gigabyte or two is liable to be killed by the memory manager. The model's weights must fit in that slice with room left for activations and the key-value cache. Compute. A mobile system-on-chip delivers a fraction of a datacenter accelerator's throughput, and it has no high-bandwidth memory feeding it, so both arithmetic and memory bandwidth are scarce. Energy and thermal. Every inference drains a battery and produces heat with nowhere to go; sustained heavy compute trips a thermal limit and the chip throttles its own clock, so a model that is fast for one query can be slow for the hundredth. No datacenter accelerators. There is no NVLink, no fast interconnect, no second device to offload to; the model runs on what the chip already has, which is a CPU, perhaps a mobile GPU, and increasingly an NPU specialized for integer matrix multiplies.

2. Stage One: Compress the Model (Borrowing Chapter 22) Intermediate

The first stage of the toolchain is to make the model small enough to fit in memory and cheap enough to run within the energy budget. These are exactly the per-node efficiency techniques developed in Chapter 22 as the baseline that distribution multiplies. Here we are not re-deriving them; we are deploying that same toolkit at the extreme end of the envelope, where the node is a phone and the multiplication factor is a billion endpoints. Three families do the work.

Quantization stores each weight in fewer bits. A parameter held as a 32-bit float can be represented, after calibration, as an 8-bit or even 4-bit integer plus a shared scale, cutting the memory footprint by four or eight times and letting the integer units of an NPU do the arithmetic. Pruning removes weights or whole structures (channels, attention heads) that contribute little, shrinking both storage and compute. Distillation trains a small student model to imitate a large teacher, so that a network small enough for the device inherits much of the behavior of one that never could be. The three compose: a model is commonly distilled to a compact architecture, pruned of dead structure, and then quantized for storage and integer execution.

The size arithmetic is direct. A model with $P$ parameters stored at $b$ bits per parameter occupies

so moving from $b = 32$ to $b = 4$ is an eightfold reduction in the weight footprint, the single largest lever available. The energy story is parallel: to a first approximation the energy per inference scales with the number of arithmetic operations and the bytes moved from memory,

where $e_{\text{op}}$ and $e_{\text{mem}}$ are the per-operation and per-byte energy costs of the chip. Integer arithmetic has a smaller $e_{\text{op}}$ than floating point, and fewer bits per weight shrinks $N_{\text{bytes}}$, so quantization attacks both terms at once. That is why it is the first reflex of on-device deployment and the lever the demo below measures directly.

The code quantizes one layer's weight matrix from 32-bit float to 8-bit integers using a single per-tensor scale, measures how much accuracy that costs, and then prints the memory footprint of the same parameter count at four common widths.

import numpy as np

rng = np.random.default_rng(0)
# A small dense layer's weight matrix: 256 x 256 fp32 parameters.
W = rng.standard_normal((256, 256)).astype(np.float32)
P = W.size

# --- Symmetric per-tensor int8 quantization (scale + zero-point) ---
qmax = 127
scale = np.max(np.abs(W)) / qmax           # one scale for the whole tensor
W_q = np.round(W / scale).astype(np.int8)  # store int8 codes
W_hat = W_q.astype(np.float32) * scale     # dequantize for inference math

mae = np.mean(np.abs(W - W_hat))                                  # reconstruction error
rel = np.linalg.norm(W - W_hat) / np.linalg.norm(W)

# --- Memory footprint of the SAME parameter count at four widths ---
def mb(bits):
    return P * bits / 8 / 1e6              # bytes = params * bits / 8

print(f"parameters            : {P:,}")
print(f"int8 scale            : {scale:.5f}")
print(f"reconstruction MAE    : {mae:.5f}")
print(f"relative L2 error     : {rel:.4%}")
print()
print(f"fp32 footprint        : {mb(32):.3f} MB")
print(f"fp16 footprint        : {mb(16):.3f} MB")
print(f"int8 footprint        : {mb(8):.3f} MB  ({mb(32)/mb(8):.0f}x vs fp32)")
print(f"int4 footprint        : {mb(4):.3f} MB  ({mb(32)/mb(4):.0f}x vs fp32)")
print()
# Scale the same widths up to a 7-billion-parameter on-device LLM.
B = 7_000_000_000
for bits, name in [(16, "fp16"), (8, "int8"), (4, "int4")]:
    print(f"7B model in {name:<4}     : {B * bits / 8 / 1e9:6.2f} GB")

parameters            : 65,536
int8 scale            : 0.03726
reconstruction MAE    : 0.00934
relative L2 error     : 1.0793%

fp32 footprint        : 0.262 MB
fp16 footprint        : 0.131 MB
int8 footprint        : 0.066 MB  (4x vs fp32)
int4 footprint        : 0.033 MB  (8x vs fp32)

7B model in fp16     :  14.00 GB
7B model in int8     :   7.00 GB
7B model in int4     :   3.50 GB

The bottom three lines of Output 34.3.1 are the whole argument for on-device quantization in miniature. A 7-billion-parameter model, a size that is now common for capable assistants, simply does not fit a phone at full or half precision, but the eightfold reduction of 4-bit quantization brings it into the memory slice the operating system will tolerate, at a relative weight error near one percent. The same calibration error that Chapter 22 analyzes as a per-node accuracy trade is, on the device, the price of admission.

3. Stage Two: Execute Through a Mobile Runtime and an NPU Intermediate

A compressed model is a file; running it on the device is the job of a mobile or edge runtime, a lightweight inference engine that loads the model, plans its execution, and dispatches each operator to the best available silicon. The runtime is what Figure 34.3.1 places between the application and the chip, and several have become standard. TensorFlow Lite (now LiteRT) targets Android and microcontrollers and pairs with quantization-aware training. ONNX Runtime Mobile runs the portable ONNX graph format with a small binary footprint. Core ML is Apple's runtime, compiling models to the Apple Neural Engine. ExecuTorch is PyTorch's on-device path, exporting a model to run without the full framework. GGML and llama.cpp brought quantized large language models to commodity CPUs and phones, and are the reason a 4-bit assistant runs on a laptop with no GPU at all.

The runtime's central trick is the delegate or execution provider: a plug-in that hands compatible subgraphs to a hardware accelerator. The most consequential accelerator on a modern device is the NPU (neural processing unit), a block of silicon built for the low-precision integer matrix multiplies that quantized inference is made of. An NPU performs an int8 matmul at a fraction of the energy a CPU spends on the float equivalent, which is why stage one (quantize) and stage two (dispatch to the NPU) are designed together: the model is quantized precisely so the NPU can run it. The DSP plays a similar role on chips without a dedicated NPU. When the runtime meets an operator no accelerator supports, it falls back to the CPU, so correctness never depends on the accelerator being present.

# TensorFlow Lite: post-training int8 quantization of a whole model.
import tensorflow as tf
conv = tf.lite.TFLiteConverter.from_saved_model("model/")
conv.optimizations = [tf.lite.Optimize.DEFAULT]          # enable quantization
conv.representative_dataset = calib_gen                  # calibrates the scales
conv.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
open("model_int8.tflite", "wb").write(conv.convert())    # 4x smaller file

# ONNX Runtime Mobile: load and run, picking the NPU/GPU delegate if present.
import onnxruntime as ort
sess = ort.InferenceSession("model.onnx",
        providers=["NnapiExecutionProvider", "CPUExecutionProvider"])
out = sess.run(None, {"input": x})                       # runs on NPU, CPU fallback

4. When the Device Is Enough, and When It Is Not Advanced

On-device inference is the right answer when the compressed model fits the envelope of Section 1 and the latency, cost, and privacy gains justify the accuracy given up to compression. That covers a great deal: wake-word detection, keyboard prediction, on-device translation, photo segmentation, and increasingly small assistant models. It contrasts sharply with the server-side path of Chapter 24, where a frontier model is sharded across many accelerators behind a continuous-batching scheduler, the operator pays for every token, and the memory budget is hundreds of gigabytes rather than a handful. The device gives up that ceiling of capability to gain locality, privacy, and zero marginal cost.

The envelope still binds, though. A model too large to quantize into the device's memory, a task that needs a frontier-scale network, or a workload whose sustained compute would overrun the thermal budget cannot live entirely on the device. The next section takes up exactly that case: when the device alone is not enough, the computation is split, with the early layers running on-device and the rest offloaded to a nearby fog node or the cloud, a hybrid that keeps the locality of on-device inference where it can and reaches for more hardware only where it must. That is the subject of Section 34.4.