Cost-Aware Distributed Experimentation

"They gave me a thousand workers and a credit-card limit. The workers were the easy part."

A Search Coordinator Watching the Meter

Every previous section of this chapter optimized for speed: how to evaluate more configurations per hour, how to stop bad trials sooner, how to keep workers from idling at a rung barrier. Speed is a means, not the end. A team that runs ten thousand trials and a team that runs one hundred can land on the same model if the second team spent its budget more wisely, and the second team will have money left over. The right accounting unit for distributed hyperparameter optimization (HPO) is therefore not the trial and not the GPU-hour but the dollar, and the right objective is the quality of the best configuration discovered for a fixed spend. This is the same cost-aware lens that Section 3.9 applied to scaling decisions and that Section 5.5 applied to energy and cost reporting; here we turn it on the search itself.

1. Search as Budget Allocation Beginner

Fix a total budget $B$, in dollars, and let a search strategy spend it across a set of trials. Each trial $t$ consumes some compute and returns a validation quality $q_t$ (say, negative validation loss, so larger is better). The search returns the best configuration it saw, with quality $q^\star = \max_t q_t$. The natural figure of merit is not how many trials ran but how much quality each dollar bought,

where $c_t$ is the dollar cost of trial $t$. Written this way, every technique in this chapter is a strategy for raising $q^\star$ under the constraint $\sum_t c_t \le B$. Smarter sampling raises the expected quality of each trial; multi-fidelity methods shrink the $c_t$ of trials that were going to lose anyway, freeing budget for more trials; cheaper compute shrinks every $c_t$ uniformly. The art of cost-aware experimentation is choosing where to push so that the constraint binds as late as possible and $q^\star$ climbs as high as possible before the money runs out.

2. Three Levers That Move the Budget Intermediate

The chapter gives three concrete levers, in increasing order of how much money they save. The first is multi-fidelity evaluation with early stopping: do not finish losers. Successive halving and ASHA (Section 21.3 and Section 21.4) spend full budget only on configurations that survive a cheap screening at low fidelity, so most trials are killed after a fraction of their cost. The dollars saved on the losers buy more configurations explored, which directly raises $q^\star$. This is the lever that turns the embarrassingly parallel breadth of search into something the budget can actually afford.

The second lever is the kind of compute you rent. Because a hyperparameter trial is short, self-contained, and restartable, it is the ideal workload for spot or preemptible instances, which clouds sell at sixty to ninety percent off the on-demand price in exchange for the right to reclaim them at any moment. A trial that is preempted is simply re-launched or resumed from its last checkpoint, exactly the elastic, fault-tolerant machinery of Section 18.6, scheduled by the cluster manager of Chapter 33. Running a search on spot capacity shrinks every $c_t$ by the same large factor, so the same dollar budget buys several times the compute.

The third lever is the most powerful for large models, and it sidesteps the search almost entirely: hyperparameter transfer. Instead of tuning the model you intend to ship, you tune a much smaller proxy and carry the winning hyperparameters across. The maximal-update parametrization (muP) makes this rigorous for neural networks by reparametrizing so that the optimal learning rate and related hyperparameters are stable as width grows; you tune at small width for pennies and transfer the result to a model hundreds of times larger. We quantify the three levers next, but their ordering is the lesson: each one saves more than the one before, and the last one can replace the search on the expensive model with a search on a cheap one.

3. Best Model per Dollar, Measured Intermediate

The reframing is only convincing with numbers, so the program below pits three strategies against the same fixed dollar budget and reports the best validation loss each one finds. Each candidate configuration has a hidden asymptotic loss; an evaluation at $r$ epochs returns that loss plus a bias and noise that both shrink as $r$ grows, so cheap low-fidelity looks are informative but unreliable. Full random search trains every sampled configuration to the full fidelity. ASHA spends full fidelity only on survivors of cheap rungs, so it explores far more configurations per dollar. ASHA on spot does the same arithmetic at the spot price, which here is roughly one third of on-demand. Lower loss is better; the figure of merit is the best loss found before the budget is exhausted.

import math, random
random.seed(11)

BUDGET_USD  = 300.0     # the fixed experiment budget
ON_DEMAND   = 3.00      # $ per GPU-hour, on-demand
SPOT        = 0.90      # $ per GPU-hour, spot / preemptible (~70% off)
EPOCH_HOURS = 0.05      # wall-clock hours to train one config for 1 epoch
R_MAX, ETA  = 27, 3     # full-fidelity epochs; ASHA halving factor

def sample_config():                       # hidden asymptotic loss
    return {"true_loss": random.uniform(0.10, 0.90)}

def observe(cfg, epochs):                  # cheap looks are biased and noisy
    gap   = 0.6 * math.exp(-epochs / 6.0)
    noise = random.gauss(0, 0.20 / math.sqrt(epochs))
    return cfg["true_loss"] + gap + noise

def cost_for(epochs, price):
    return epochs * EPOCH_HOURS * price

def full_random(price):                    # train every config to R_MAX
    spent, best, trials = 0.0, math.inf, 0
    while spent + cost_for(R_MAX, price) <= BUDGET_USD:
        spent += cost_for(R_MAX, price)
        best   = min(best, observe(sample_config(), R_MAX)); trials += 1
    return best, trials, spent

def asha(price):                           # full fidelity only for survivors
    rungs = [ETA**k for k in range(int(math.log(R_MAX, ETA)) + 1)]
    n0 = ETA ** (len(rungs) - 1)
    def bracket_cost():
        c, n = 0.0, n0
        for r in rungs: c += cost_for(r, price) * n; n = max(1, n // ETA)
        return c
    bc, spent, best, trials = bracket_cost(), 0.0, math.inf, 0
    while spent + bc <= BUDGET_USD:
        alive = [sample_config() for _ in range(n0)]; trials += len(alive)
        for r in rungs:
            spent += cost_for(r, price) * len(alive)
            alive = sorted(alive, key=lambda c: observe(c, r))[:max(1, len(alive)//ETA)]
            for c in alive: best = min(best, observe(c, r))
    return best, trials, spent

rows = [("full random (on-demand)", *full_random(ON_DEMAND)),
        ("ASHA (on-demand)",        *asha(ON_DEMAND)),
        ("ASHA on spot",            *asha(SPOT))]
print(f"{'strategy':<26}{'best loss':>11}{'configs':>10}{'$/quality':>12}")
for name, best, trials, spent in rows:
    cpq = spent / max(1e-6, 1.0 - best)    # dollars per unit of quality
    print(f"{name:<26}{best:>11.4f}{trials:>10}{cpq:>12.2f}")
print(f"\nconfigs explored, ASHA-on-spot vs full random : "
      f"{rows[2][2] / rows[0][2]:.1f}x more")

strategy                    best loss   configs   $/quality
full random (on-demand)        0.0851        74      327.57
ASHA (on-demand)               0.0836       486      318.20
ASHA on spot                   0.0596      1647      315.26

configs explored, ASHA-on-spot vs full random : 22.3x more

The numbers make the case Figure 21.8.1 drew. Early stopping alone (the jump from row one to row two) multiplies the configurations explored without raising the spend, because the budget no longer drains into trials that were never going to win. Spot pricing (row three) multiplies the affordable compute again, and the extra exploration converts directly into a better model. Cost-per-quality, the dollars spent per unit of $1 - \text{loss}$, falls across the rows even though all three nearly exhaust the budget: the cost-aware search simply buys more quality with the same money.

# Ray Tune: ASHA early stopping + spot retries, the cost-aware combo in ~10 lines.
from ray import tune
from ray.tune.schedulers import ASHAScheduler

tuner = tune.Tuner(
    train_fn,
    tune_config=tune.TuneConfig(
        scheduler=ASHAScheduler(metric="val_loss", mode="min",
                                max_t=27, grace_period=1, reduction_factor=3),
        num_samples=2000),                 # propose many; ASHA kills losers early
    run_config=tune.RunConfig(
        storage_path="s3://my-bucket/hpo", # checkpoints survive spot preemption
        failure_config=tune.FailureConfig(max_failures=-1)))  # retry preempted trials
results = tuner.fit()
print(results.get_best_result(metric="val_loss", mode="min").config)

4. Diminishing Returns and When to Stop Advanced

The last cost-aware skill is knowing when to stop searching. The quality curve of Figure 21.8.1 is concave: the first dollars buy large gains as the search escapes obviously bad regions, and later dollars buy progressively less as it polishes an already good configuration. The rational stopping point is where the marginal quality per dollar, $\mathrm{d}q^\star / \mathrm{d}B$, falls below what that dollar is worth to you elsewhere, whether that is a different experiment, a larger training run, or simply not spent. Continuing past that point is the most common waste in practice: a search that has plateaued still bills for every trial. A simple, robust rule is to stop when the best quality has not improved by more than a small threshold over the last several brackets, which is the search-level analogue of the per-trial early stopping that ASHA already applies inside a bracket.

Part IV ends here. We began the part able to compute an exact gradient on one machine and now can train, shard, route, recover, and tune models that no single device could hold, always under an explicit cost budget. The models exist; the next problem is making them answer. Part V turns to distributed inference and serving, and it opens by establishing the per-node efficiency that the whole serving fleet multiplies, in Chapter 22.