On-Device Inference

"They quantized me to four bits and dropped me into 4 GB of phone RAM. I lost a little precision and gained a billion homes."

A Quantized Model, Living in 4 GB of Phone RAM

Every form of distribution in this book so far has moved work between machines an operator owns: shards across a cluster, replicas across a serving fleet, agents across a network. On-device inference moves the model to the one machine the operator does not own, the user's own hardware, and then runs the whole computation there. The motivation is the same triad that pushed work off a single machine in Section 1.1, read in reverse: when the endpoints number in the billions, sending every query to a datacenter is the ceiling. Latency crosses a wireless link twice, privacy-sensitive data leaves the device, the service pays for every inference, and a dropped connection means no answer at all. Running the model locally removes all four problems at once, in exchange for living inside an envelope far smaller than any node in Chapter 24 ever sees.

1. Four Constraints That Have No Datacenter Remedy Beginner

A datacenter node answers resource pressure by adding more of the resource: more memory, more accelerators, more power and cooling. The device cannot. Its constraints are fixed at manufacture and shared with everything else the user is doing, so the model must be shaped to fit them rather than the other way around. Four of them bind, and they bind together.

Memory. A phone may report 8 GB of RAM, but the operating system, the foreground app, and the graphics surface claim most of it; an inference engine that demands more than a gigabyte or two is liable to be killed by the memory manager. The model's weights must fit in that slice with room left for activations and the key-value cache. Compute. A mobile system-on-chip delivers a fraction of a datacenter accelerator's throughput, and it has no high-bandwidth memory feeding it, so both arithmetic and memory bandwidth are scarce. Energy and thermal. Every inference drains a battery and produces heat with nowhere to go; sustained heavy compute trips a thermal limit and the chip throttles its own clock, so a model that is fast for one query can be slow for the hundredth. No datacenter accelerators. There is no NVLink, no fast interconnect, no second device to offload to; the model runs on what the chip already has, which is a CPU, perhaps a mobile GPU, and increasingly an NPU specialized for integer matrix multiplies.

2. Stage One: Compress the Model (Borrowing Chapter 22) Intermediate

The first stage of the toolchain is to make the model small enough to fit in memory and cheap enough to run within the energy budget. These are exactly the per-node efficiency techniques developed in Chapter 22 as the baseline that distribution multiplies. Here we are not re-deriving them; we are deploying that same toolkit at the extreme end of the envelope, where the node is a phone and the multiplication factor is a billion endpoints. Three families do the work.

Quantization stores each weight in fewer bits. A parameter held as a 32-bit float can be represented, after calibration, as an 8-bit or even 4-bit integer plus a shared scale, cutting the memory footprint by four or eight times and letting the integer units of an NPU do the arithmetic. Pruning removes weights or whole structures (channels, attention heads) that contribute little, shrinking both storage and compute. Distillation trains a small student model to imitate a large teacher, so that a network small enough for the device inherits much of the behavior of one that never could be. The three compose: a model is commonly distilled to a compact architecture, pruned of dead structure, and then quantized for storage and integer execution.

The size arithmetic is direct. A model with $P$ parameters stored at $b$ bits per parameter occupies

so moving from $b = 32$ to $b = 4$ is an eightfold reduction in the weight footprint, the single largest lever available. The energy story is parallel: to a first approximation the energy per inference scales with the number of arithmetic operations and the bytes moved from memory,

where $e_{\text{op}}$ and $e_{\text{mem}}$ are the per-operation and per-byte energy costs of the chip. Integer arithmetic has a smaller $e_{\text{op}}$ than floating point, and fewer bits per weight shrinks $N_{\text{bytes}}$, so quantization attacks both terms at once. That is why it is the first reflex of on-device deployment and the lever the demo below measures directly.

The code quantizes one layer's weight matrix from 32-bit float to 8-bit integers using a single per-tensor scale, measures how much accuracy that costs, and then prints the memory footprint of the same parameter count at four common widths.

import numpy as np

rng = np.random.default_rng(0)
# A small dense layer's weight matrix: 256 x 256 fp32 parameters.
W = rng.standard_normal((256, 256)).astype(np.float32)
P = W.size

# --- Symmetric per-tensor int8 quantization (scale + zero-point) ---
qmax = 127
scale = np.max(np.abs(W)) / qmax           # one scale for the whole tensor
W_q = np.round(W / scale).astype(np.int8)  # store int8 codes
W_hat = W_q.astype(np.float32) * scale     # dequantize for inference math

mae = np.mean(np.abs(W - W_hat))                                  # reconstruction error
rel = np.linalg.norm(W - W_hat) / np.linalg.norm(W)

# --- Memory footprint of the SAME parameter count at four widths ---
def mb(bits):
    return P * bits / 8 / 1e6              # bytes = params * bits / 8

print(f"parameters            : {P:,}")
print(f"int8 scale            : {scale:.5f}")
print(f"reconstruction MAE    : {mae:.5f}")
print(f"relative L2 error     : {rel:.4%}")
print()
print(f"fp32 footprint        : {mb(32):.3f} MB")
print(f"fp16 footprint        : {mb(16):.3f} MB")
print(f"int8 footprint        : {mb(8):.3f} MB  ({mb(32)/mb(8):.0f}x vs fp32)")
print(f"int4 footprint        : {mb(4):.3f} MB  ({mb(32)/mb(4):.0f}x vs fp32)")
print()
# Scale the same widths up to a 7-billion-parameter on-device LLM.
B = 7_000_000_000
for bits, name in [(16, "fp16"), (8, "int8"), (4, "int4")]:
    print(f"7B model in {name:<4}     : {B * bits / 8 / 1e9:6.2f} GB")

parameters            : 65,536
int8 scale            : 0.03726
reconstruction MAE    : 0.00934
relative L2 error     : 1.0793%

fp32 footprint        : 0.262 MB
fp16 footprint        : 0.131 MB
int8 footprint        : 0.066 MB  (4x vs fp32)
int4 footprint        : 0.033 MB  (8x vs fp32)

7B model in fp16     :  14.00 GB
7B model in int8     :   7.00 GB
7B model in int4     :   3.50 GB

The bottom three lines of Output 34.3.1 are the whole argument for on-device quantization in miniature. A 7-billion-parameter model, a size that is now common for capable assistants, simply does not fit a phone at full or half precision, but the eightfold reduction of 4-bit quantization brings it into the memory slice the operating system will tolerate, at a relative weight error near one percent. The same calibration error that Chapter 22 analyzes as a per-node accuracy trade is, on the device, the price of admission.

3. Stage Two: Execute Through a Mobile Runtime and an NPU Intermediate

A compressed model is a file; running it on the device is the job of a mobile or edge runtime, a lightweight inference engine that loads the model, plans its execution, and dispatches each operator to the best available silicon. The runtime is what Figure 34.3.1 places between the application and the chip, and several have become standard. TensorFlow Lite (now LiteRT) targets Android and microcontrollers and pairs with quantization-aware training. ONNX Runtime Mobile runs the portable ONNX graph format with a small binary footprint. Core ML is Apple's runtime, compiling models to the Apple Neural Engine. ExecuTorch is PyTorch's on-device path, exporting a model to run without the full framework. GGML and llama.cpp brought quantized large language models to commodity CPUs and phones, and are the reason a 4-bit assistant runs on a laptop with no GPU at all.

The runtime's central trick is the delegate or execution provider: a plug-in that hands compatible subgraphs to a hardware accelerator. The most consequential accelerator on a modern device is the NPU (neural processing unit), a block of silicon built for the low-precision integer matrix multiplies that quantized inference is made of. An NPU performs an int8 matmul at a fraction of the energy a CPU spends on the float equivalent, which is why stage one (quantize) and stage two (dispatch to the NPU) are designed together: the model is quantized precisely so the NPU can run it. The DSP plays a similar role on chips without a dedicated NPU. When the runtime meets an operator no accelerator supports, it falls back to the CPU, so correctness never depends on the accelerator being present.

# TensorFlow Lite: post-training int8 quantization of a whole model.
import tensorflow as tf
conv = tf.lite.TFLiteConverter.from_saved_model("model/")
conv.optimizations = [tf.lite.Optimize.DEFAULT]          # enable quantization
conv.representative_dataset = calib_gen                  # calibrates the scales
conv.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
open("model_int8.tflite", "wb").write(conv.convert())    # 4x smaller file

# ONNX Runtime Mobile: load and run, picking the NPU/GPU delegate if present.
import onnxruntime as ort
sess = ort.InferenceSession("model.onnx",
        providers=["NnapiExecutionProvider", "CPUExecutionProvider"])
out = sess.run(None, {"input": x})                       # runs on NPU, CPU fallback

4. When the Device Is Enough, and When It Is Not Advanced

On-device inference is the right answer when the compressed model fits the envelope of Section 1 and the latency, cost, and privacy gains justify the accuracy given up to compression. That covers a great deal: wake-word detection, keyboard prediction, on-device translation, photo segmentation, and increasingly small assistant models. It contrasts sharply with the server-side path of Chapter 24, where a frontier model is sharded across many accelerators behind a continuous-batching scheduler, the operator pays for every token, and the memory budget is hundreds of gigabytes rather than a handful. The device gives up that ceiling of capability to gain locality, privacy, and zero marginal cost.

The envelope still binds, though. A model too large to quantize into the device's memory, a task that needs a frontier-scale network, or a workload whose sustained compute would overrun the thermal budget cannot live entirely on the device. The next section takes up exactly that case: when the device alone is not enough, the computation is split, with the early layers running on-device and the rest offloaded to a nearby fog node or the cloud, a hybrid that keeps the locality of on-device inference where it can and reaches for more hardware only where it must. That is the subject of Section 34.4.

5. On-Device Generative LLMs Advanced

The sections above treated on-device inference as the domain of compact convolutional nets and keyword spotters. Since 2024, that picture has changed substantially. Mobile system-on-chips now integrate dedicated neural processing units delivering 30 to 50 TOPS at 5 to 10 W, and flagship phones with 8 to 16 GB of RAM can run 4-bit-quantised 7-to-8-billion-parameter language models at 30 to 90 tokens per second. The device has become a credible host for generative intelligence, not merely for classification or keyword spotting, and a new stack has grown up to exploit that shift.

5.1 Why It Is Possible Now: Mobile NPUs

Three hardware generations made on-device generative inference practical. Apple's Neural Engine in the A17 and A18 chips delivers 35 to 38 TOPS at under 5 W; Qualcomm's Hexagon NPU in the Snapdragon 8 Gen 3 reaches 45 TOPS; MediaTek's APU in the Dimensity 9300 delivers 33 TOPS. These figures matter because a generative decoder step is a sequence of matrix-vector products over the weight matrix: each step reads the full model weights from memory once, which is precisely the operation an integer-optimised NPU executes most efficiently. At 40 TOPS of 4-bit throughput, a 7B model's 3.5 GB of weights can be consumed in roughly a second of arithmetic, which translates to tens of decode tokens per second on a device that consumes less power than a reading lamp.

The memory-bandwidth constraint is equally important. A 7B model at 4-bit precision occupies 3.5 GB (from $P \cdot b/8 = 7 \times 10^9 \times 4 / 8 = 3.5 \times 10^9$ bytes). A phone with 12 GB/s of LPDDR5X memory bandwidth can stream those weights in roughly $3.5 \times 10^9 / (12 \times 10^9) \approx 0.29$ seconds per decode pass, yielding a theoretical ceiling of about 3.5 tokens per second from memory bandwidth alone, before the NPU's arithmetic adds its own ceiling. In practice, grouped quantization and NPU instruction packing push observed rates to 30 to 90 tokens per second on current hardware, meaning the NPU arithmetic is fast enough that the bottleneck shifts back to memory bandwidth, which is exactly the decode-phase regime described in Chapter 24.

5.2 The Quantisation and Runtime Stack

Section 2 showed that a 7B model at int4 occupies 3.5 GB. On-device generative inference requires two additional refinements beyond the symmetric per-tensor scheme of Code 34.3.1: weight-only quantization with grouped scales, and a runtime that compiles the resulting model to the device NPU.

Weight-only quantisation (AWQ and GPTQ). Activation-aware Weight Quantization (AWQ) and Generative Pre-trained Transformer Quantization (GPTQ) both quantize only the weights, leaving activations in floating point. This avoids accumulating quantization error through multiple activation non-linearities, which would degrade generation quality. AWQ selects the scale for each weight group by minimizing the reconstruction error on a small calibration set of activations, while GPTQ solves a layer-wise second-order minimization. Both produce 4-bit weights with per-group scales.

Grouped quantisation. Rather than one scale per tensor or per channel, grouped quantisation assigns one scale to every $g$ consecutive weights, typically $g = 128$. Each scale is stored in float16, so the overhead is $P / g$ float16 values added to $P$ int4 values. At $g = 128$, the overhead is 0.78 percent of the weight footprint, a negligible price for the quality preserved by using a tighter local scale. The total memory for a 7B model at 4-bit with $g = 128$ groups remains comfortably below 4 GB.

Runtimes. Three runtimes are in wide use as of 2025 to 2026. llama.cpp compiles quantized GGUF-format models to CPU, GPU, and Metal backends with a single binary; it was the first runtime to run a 7B model on consumer hardware. ExecuTorch 1.0 (GA October 2025) is PyTorch's on-device path: a model is exported from PyTorch, quantized, and compiled to a backend-specific delegate, with a Hexagon NPU backend that achieves 18 to 38 times faster prefill than CPU-only execution. MLC-LLM uses the TVM compiler to generate optimised kernels for Metal, OpenCL, and Vulkan, covering the full range of mobile and embedded GPUs.

5.3 The Prefill/Decode Split and Hybrid NPU Scheduling

Generative inference has two phases with very different arithmetic characters, as Chapter 24 established. Prefill processes the input prompt in parallel: it is a batch of large matrix-matrix products (the query, key, and value projections over all prompt tokens), which saturates GPU parallelism but is irregular and memory-intensive on an NPU. Decode generates one token at a time: it is a sequence of matrix-vector products over the weight matrix, which an NPU executes extremely efficiently because the access pattern is regular and the arithmetic is integer.

This asymmetry motivates hybrid CPU-NPU scheduling. The prefill phase is routed to the CPU plus GPU, where multi-core parallelism and the GPU's floating-point throughput handle the irregular attention computation efficiently. The decode phase is routed to the NPU, which sustains integer matmul throughput at low power. ExecuTorch's Hexagon backend implements exactly this split, reporting 18 to 38 times faster prefill than CPU-only in the case where it routes the attention computation to GPU subgraphs and the linear layers to the Hexagon NPU. The practitioner's rule of thumb: use the NPU for weight-reads-and-accumulate (the linear layers in decode), use the GPU or CPU for irregular attention and activation operations (prefill).

5.4 Exporting a Model for On-Device Deployment with ExecuTorch

The pipeline for deploying a quantized LLM to a device with ExecuTorch follows a three-stage pattern: export the model from PyTorch to a portable representation, apply weight-only quantization, and compile the result to a device-specific backend. The snippet below shows the canonical export pipeline. The resulting .pte file is the artifact shipped to the device runtime.

import torch
from executorch.exir import to_edge
from torch.export import export
from torchao.quantization import quantize_, int4_weight_only

# 1. Load (or define) the model in eager mode.
model = load_pretrained_llm("meta-llama/Llama-3.2-3B")  # 3B weights, float32
model.eval()

# 2. Apply 4-bit weight-only quantisation with group size 128.
#    AWQ/GPTQ calibration would precede this step in production.
quantize_(model, int4_weight_only(group_size=128))

# 3. Export to a strict portable graph (torch.export captures the compute graph).
example_inputs = (torch.zeros(1, 64, dtype=torch.long),)  # (batch, seq_len)
exported = export(model, example_inputs)

# 4. Lower to the ExecuTorch edge dialect, then compile to the Hexagon backend.
edge_prog = to_edge(exported)
et_prog = edge_prog.to_executorch()          # backend-agnostic .pte bytecode

# 5. Serialize the artifact for on-device deployment.
with open("llm_q4.pte", "wb") as f:
    et_prog.write_to_file(f)

print("Exported model size:", round(f.tell() / 1e9, 2), "GB")
# Expected for a 3B model at int4: approx 1.7 GB

# Download a pre-quantized 7B model in Q4_K_M format from HuggingFace.
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
    --include "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf" --local-dir ./models

# Run it on CPU (or Metal GPU on macOS via -ngl 99).
llama-cli -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
    -p "Explain on-device AI in one paragraph." -n 200

5.5 Tradeoffs: Privacy, Offline Operation, and Capability Constraints

On-device generative inference offers three advantages that cloud serving cannot match at any price. Privacy: the input prompt and all intermediate activations remain on the device; no token is transmitted to any server. This matters for personal writing, health queries, private messages, and any scenario where the user would not want a cloud provider to see the content. Offline operation: the model answers regardless of network connectivity, which is essential for field devices, aircraft, and low-bandwidth environments. Zero marginal cost: each query costs the operator nothing, because the compute is the user's own hardware.

Three constraints push back. Context window is limited by RAM: the key-value cache for a 7B model at context length $L$ occupies $2 \cdot L \cdot n_{\text{layers}} \cdot d_{\text{head}} \cdot n_{\text{heads}} \cdot 2$ bytes (two for float16, two tensors for K and V). For a typical 32-layer, 4096-hidden-dim, 32-head model at $L = 4096$ tokens: $2 \times 4096 \times 32 \times 128 \times 32 \times 2 \approx 2.1$ GB, which added to the 3.5 GB of weights fills 8 GB RAM completely. In practice, phones support 2048 to 4096 token contexts; 8K or longer contexts are not feasible without offloading. Model quality is capped by size: a 3.5 GB model is at most a 7B-parameter model, and frontier capability requires 70B or more parameters. Complex reasoning, long-document summarisation, and multi-step tool use remain cloud tasks. Battery life constrains sustained inference: running an NPU at 40 TOPS continuously at 10 W draws a 4000 mAh battery down in roughly 4 hours of uninterrupted inference, so on-device LLMs are appropriate for interactive, bursty workloads rather than continuous background processing.