Calculate the outage tax before your next GPU cluster expansion

The trade-off is blunt: add redundant GPU capacity and pay for idle power, or accept failover latency and pay in SLA credits. The arithmetic comes before the opinion. Outage tax = compute burn (GPU $/hr x cluster size x expected MTTR) + redundant power $/month + standby capacity $/month + failover penalty - SLA credits/revenue saved. If outage tax > smaller, faster-recovering cluster cost, you are buying availability you do not need, and the smaller cluster wins; if outage tax < smaller, faster-recovering cluster cost, the expansion is a reliability purchase, not a capacity decision.

A utility outage affected many customers and was later restored; use that event as the MTTR/SLA-credit anchor for the formula. External power can fail in ways your cluster cannot fully prevent, and the cost of that failure is not only the down time.

An exam center in Jaipur had internal power failures during exams, attributed to unknown local electrical issues. For an ML platform, map the interruption to the four cost lines: degraded model routing is the standby capacity term, stale cache is the compute burn term, drained queues are the failover penalty, and support load is the penalty term.

Costs

Break the outage tax into four cost lines. The first is compute burn during recovery: GPUs that are powered but not serving, plus the time to warm caches, reload weights, and restore traffic. The second is redundant power: extra feeders, breakers, UPS capacity, and cooling headroom that exist only to survive a fault. The third is standby capacity: idle GPUs, spare nodes, or a second region that must be kept current enough to take traffic. The fourth is the penalty term: failed requests, SLA credits, lost subscriptions, and the engineering time spent proving the incident was not your fault.

Do not treat these as separate budgets. They interact. More standby capacity can reduce MTTR, but it raises the monthly floor. More redundant power can reduce the chance of a full site loss, but it can also hide a failing component until the failure is worse. The formula forces the comparison. If your expected MTTR is long, the compute burn term dominates. If your standby fleet is large, the monthly floor dominates. If your SLA credits are small, the revenue saved term is weak, and the outage tax looks worse than your dashboard suggests.

Serving

Serving architecture decides how much of the outage tax you can avoid. A single data center with one power path has a simple failure mode: the path fails, and your recovery time is the time to restore power or reroute traffic. A multi-path design can reduce that time, but it costs more in power, cooling, and operational complexity. The right answer depends on the penalty term, not on the desire for a clean architecture diagram.

Use three numeric checks before expanding. First, measure failover latency under a real power-loss simulation, not a network partition. If the cluster cannot detect loss, drain traffic, and restore service quickly, the standby capacity is not reducing outage tax; it is just sitting there. Second, price the standby fleet as a monthly power/cooling line item, including power and cooling, not just GPU rental. Third, compare the failover latency penalty against the cost of keeping a smaller cluster with faster recovery. If the smaller cluster recovers in a time that avoids most SLA credits, the larger cluster may be paying for availability that never gets used.

For inference SREs, the practical rule is to treat power as a first-class dependency. Track power events, breaker trips, UPS transitions, and cooling faults with the same severity as GPU failures. If a power event can take down a serving pool, it belongs in the outage tax calculation. If it cannot, document why, and keep the redundancy budget small enough that it does not distort the inference cost.

Models

Model choice changes the outage tax in ways that are easy to miss. A large model may need more GPUs, longer warm-up, and more memory to reload, which raises the compute burn term. A smaller or quantized model may serve the same traffic with less capacity, shorter recovery, and lower standby cost. The trade-off is not only accuracy. It is the cost of being unavailable while the model is being brought back.

Before the next expansion, run the formula with your actual serving stack. Use the real GPU rate, the real cluster size, the measured MTTR, and the real SLA credit exposure. If the outage tax is high, ask whether the fix is more capacity or faster recovery. More capacity is a blunt instrument. Faster recovery is a sharper one: better health checks, cleaner failover, smaller blast radius, and a standby path that can actually take traffic. The goal is not to eliminate every power failure. The goal is to make the cost of the next failure a number you can defend, not a surprise after the fact.