Edge AI as Distribution to the Periphery

"Every frame, I face the same question: solve it here in my own small silicon, or pay the network to ask someone smarter. Most days I decide it is cheaper to think for myself."

A Sensor at the Edge, Deciding Whether to Phone Home

The datacenter of Chapter 33 is the natural home of scale-out as we have developed it. Thousands of near-identical accelerators sit in racks, joined by an interconnect engineered for bandwidth, powered and cooled without limit, scheduled by a system that assumes every node is reachable and roughly equal. Data flows in, gradients and predictions flow out, and the whole arrangement behaves as one large coherent machine. That picture is correct, and it underwrites most of modern AI training and a great deal of its serving. It also quietly assumes that the data has already arrived at the center, that moving every input to the cluster and every output back is affordable. For an enormous class of real systems that assumption is false, and the failure of that single assumption is what creates the field of edge AI.

Consider where data actually originates. A self-driving car generates terabytes of sensor readings per hour, on a road, miles from any datacenter, on a cellular link that is metered and intermittent. A factory inspection camera produces a video stream that must trigger a reject decision within milliseconds, faster than light can make the round trip to a regional cloud. A medical wearable records signals whose owner has not consented to streaming them anywhere. A smart speaker must keep working when the home network drops. In each case the data is born at the periphery, and the question is no longer "how do we bring it to the compute" but "how do we bring the compute to it". That inversion, distribution to the periphery rather than concentration at the center, is the subject of this chapter.

1. The Center of Gravity Was Always a Choice Beginner

It is worth seeing clearly that pulling data to a central cluster was never a law of nature; it was an economic default that held while bandwidth was cheap relative to compute and while latency to the cloud was tolerable. Under those conditions the simplest correct design is to ship every input to a place with abundant accelerators, run the model there, and ship the answer back. That default produced the architectures of Parts II through V, and it remains right whenever the data is already in the datacenter or is cheap to move there. Edge AI is what you reach for when the default breaks, and it breaks along four fairly distinct seams that are useful to keep separate because, exactly as with the three ceilings of Section 1.1, each one calls for a different response.

The first seam is latency. The round trip from a device to a regional datacenter and back is bounded below by physics and inflated by every queue and protocol hop along the way; tens of milliseconds is a good case, and many control loops (a robot arm, a vehicle, an augmented-reality overlay) need an answer faster than that round trip can possibly deliver. The second is bandwidth and its cost. A fleet of high-resolution cameras or a population of always-listening microphones generates far more raw data than any network can carry to the center, and on a metered cellular or satellite link the bytes are billed. The third is privacy and regulation. Health signals, faces, keystrokes, and location traces are exactly the data that users and law are least willing to see streamed to a third party, so the safest place to process them is the device that already holds them. The fourth is autonomy under intermittent connectivity: a tractor in a field, a drone past the horizon, or a phone in a tunnel must keep making decisions when the link to the center is slow, expensive, or simply gone.

2. The Cloud-to-Device Continuum Beginner

It is a common simplification to speak of "the cloud" and "the edge" as two boxes, but real deployments live on a continuum with at least four named tiers, drawn in Figure 34.1.1 and summarized in Table 34.1.1. At one end is the cloud: the datacenter cluster of Chapter 33, with effectively unlimited compute, energy, and memory, holding the most aggregated data but sitting farthest, in latency and in bytes, from where any single observation is made. At the other end is the device itself, the phone or camera or microcontroller that captures the data, with the least compute and energy but zero network latency to its own sensors and the ability to keep working when fully disconnected. Between them sit fog and edge tiers: regional gateways, on-premises servers, and base-station compute that trade a little proximity for a lot more capability, aggregating many devices without making the full trip to the cloud.

The design question is therefore not "cloud or edge?" but "which piece of which workload belongs on which tier?". A speech assistant might run wake-word detection on the device (it must always be on and must not stream audio), short-command recognition at the edge, and a full conversational model in the cloud. A camera network might run motion and person detection on each camera, aggregate and re-identify across cameras at a fog gateway, and retain only flagged events centrally. The skill this chapter builds is reading a workload's dominant constraint, latency, bandwidth, privacy, or autonomy, and placing each stage at the tier where that constraint is cheapest to satisfy. We give the fog tier its own treatment in Section 34.2 and the device tier in Section 34.3.

3. The Latency and Bandwidth Budget, Made Explicit Intermediate

The drivers become concrete the moment we write them as a budget. Consider the end-to-end latency of answering one query when the model lives in the cloud. The data must serialize onto the uplink, traverse the network there and back, wait in the server's queue, run through the model, and return:

Here $B_{\uparrow}$ and $B_{\downarrow}$ are the bytes sent up and down, $R_{\uparrow}$ and $R_{\downarrow}$ the link rates, and $T_{\text{rtt}}$ the network round-trip time. Running the same query on the device deletes the entire network term, leaving only local compute:

The edge pays for this with a slower compute term, because the on-device model is smaller and the silicon weaker, so placement is a genuine trade rather than a free win: the edge wins precisely when the network terms it removes exceed the extra compute it adds. The bandwidth picture is starker still, because it accumulates. If the device streams every observation to the cloud it sends $B_{\uparrow}$ bytes per query; if it runs the model locally and emits only the decision it sends a few bytes per query (or none, if it acts locally), and over a busy day the ratio of the two is enormous. The demo in Code 34.1.1 plugs representative numbers for a 15-frame-per-second camera into both equations.

def serialize_ms(nbytes, mbps):
    # time to push nbytes onto a link of the given rate, in milliseconds
    return (nbytes * 8) / (mbps * 1e6) * 1e3

# One 224x224 camera frame, JPEG-compressed before upload.
jpeg_bytes, label_bytes = 30_000, 200
rtt_ms, up_mbps, down_mbps = 50.0, 8.0, 30.0      # a typical 4G uplink

# Cloud path: T_rtt + serialize up + compute + serialize down.
cloud_total = (rtt_ms
               + serialize_ms(jpeg_bytes, up_mbps)
               + 8.0                                # big model, fast once data lands
               + serialize_ms(label_bytes, down_mbps))

# Edge path: only local compute, no network term at all.
edge_total = 35.0                                   # smaller model on a mobile NPU

# Bandwidth over a day at 15 fps: stream every frame vs. send only decisions.
frames = 15 * 86_400
cloud_mb = frames * jpeg_bytes / 1e6
edge_mb  = frames * label_bytes / 1e6
print(f"cloud latency {cloud_total:6.1f} ms   edge latency {edge_total:6.1f} ms")
print(f"cloud data {cloud_mb:8.1f} MB/day   edge data {edge_mb:6.1f} MB/day")
print(f"latency speedup {cloud_total/edge_total:4.1f}x   bandwidth cut {cloud_mb/edge_mb:4.0f}x")

cloud latency   88.1 ms   edge latency   35.0 ms
cloud data  38880.0 MB/day   edge data  259.2 MB/day
latency speedup  2.5x   bandwidth cut  150x

Two lessons fall out of Output 34.1.1. First, the edge wins on latency only because the network terms it removes (here 80 of the cloud's 88 milliseconds) dwarf the extra on-device compute; if the model were heavy enough to make $T_{\text{compute}}^{\text{edge}}$ exceed the whole cloud round trip, the trade would flip, which is why model compression and small-model design are inseparable from edge deployment and occupy much of Section 34.3. Second, the bandwidth saving does not depend on any latency assumption at all; it follows purely from sending decisions instead of raw observations, and it is what makes large camera and sensor fleets economically possible. The full machinery for measuring these quantities rigorously, tail latency, energy per inference, accuracy under compression, lives in Chapter 5; here we use them only to motivate placement.

4. A Different Regime of Distribution Intermediate

The datacenter cluster of Chapter 33 and an edge fleet are both distributed systems, but almost every assumption that made the cluster tractable is reversed at the periphery, and naming the reversals is the fastest way to see why edge AI needs its own chapter. In the cluster the nodes are homogeneous; at the edge they span a decade of hardware generations, operating systems, and accelerators, so the same model must run acceptably on a flagship phone and a five-year-old budget device. In the cluster the network is fast, private, and engineered for collective communication; at the edge it is slow, metered, shared with the public internet, and frequently absent, so the tight synchronous all-reduce of Chapter 15 is simply not available. In the cluster a node either works or is quickly replaced by a scheduler that sees the whole fleet; at the edge a node may be powered off by its owner, run out of battery, or roam out of coverage, and no central scheduler controls it.

The scale itself is different in kind. A large training cluster has thousands of nodes; a consumer edge deployment has tens or hundreds of millions, each contributing a sliver of data the center may never even see. Resources are tight and non-negotiable: a phone budgets compute against battery and thermal limits, a microcontroller against kilobytes of RAM, and exceeding either budget is not a slowdown but a dead device or a drained battery. These properties, heterogeneity, unreliable and intermittent connectivity, no central control, extreme node count, and hard resource limits, are not nuisances layered on top of ordinary distribution; they define a distinct regime in which familiar primitives must be rebuilt. Federated learning, the subject of Section 34.6, is precisely what data-parallel training becomes once you accept all of these constraints at once, and it is the direct descendant of the FedAvg and gossip methods built in Chapter 14.

# Export once from your training framework, then deploy per target runtime.
import torch
torch.onnx.export(model, example_input, "detector.onnx",   # portable graph
                  opset_version=17, dynamic_axes={"input": {0: "batch"}})

# On a server / edge box: ONNX Runtime picks the best local provider.
import onnxruntime as ort
sess = ort.InferenceSession("detector.onnx",
                            providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
labels = sess.run(None, {"input": frame})            # one call, runtime handles the hardware

# On a phone you would instead convert to TensorFlow Lite / Core ML; on a
# microcontroller, to TFLite Micro; in a browser, to ONNX Runtime Web or WebNN.

5. What This Chapter Builds, and Where It Connects Beginner

With the thesis set and the continuum mapped, the rest of the chapter develops each tier and the learning problem that spans them. Section 34.2 studies the fog tier, the regional gateways and on-premises servers that aggregate many devices without the full trip to the cloud, and the placement problem of deciding which stage of a pipeline runs where. Section 34.3 descends to the device itself, where model compression, quantization, and small-model design make inference fit inside a battery and a few milliwatts. Later sections build the fleet machinery (over-the-air model delivery, monitoring devices you do not control) and culminate in Section 34.6, federated learning at the edge, where the devices stop merely running a model and start collectively training one, carrying the FedAvg and decentralized methods of Chapter 14 into the harsh connectivity and heterogeneity regime named in Section 4.

Throughout, the evaluation discipline of Chapter 5 is what keeps placement decisions honest. "Run it on the edge" is a hypothesis about latency, energy, accuracy, and cost that must be measured, not asserted, because the on-device compute term can quietly erase the network saving, because quantization can quietly erase accuracy, and because the tail latency that matters to a user is not the average that a quick benchmark reports. Edge AI is where the trade-offs of the whole book, the communication tax of Section 1.1, the cost models of Chapter 3, the reliability concerns that thread through Part VII, become most visible, because here the machines are the ones in people's pockets and on the roads, and the constraints are real, physical, and unforgiving.