Cut AI Outage Cost: SLOs, Fallbacks, Downtime Budgets

downtime_cost = duration_min × revenue_per_min × failure_fraction; if downtime_cost > downtime_budget, the dependency fails. The Microsoft issue is the test case: user reports of Outlook problems spiked on Downdetector, and authentication was affected. A serving dependency that can fail is a costed surface, not a neutral upstream. The fix is not more dashboards. It is a dependency map, an SLO, a fallback, a downtime budget, and a drill that proves the fallback works.

Trace the Microsoft issue as one feature path, checking each state against downtime_budget. Authentication misconfiguration: a possible misconfiguration prevented authentication components from deploying as expected, so authentication is the dependency to map and the state to check against downtime_budget. Multi-service impact: the issue affected Exchange Online and some other services. The reported impact extended across multiple Microsoft 365 services. That user path is the SLO surface and the state to check against downtime_budget. Partial restoration: mailbox connectivity returned to normal while search restoration continued, so the degraded fallback must keep downtime_cost under downtime_budget.

Map the dependency before you set the SLO

Start with a dependency map that names the failure surface, not the vendor. For each AI serving path, list the dependencies that can block a request: authentication, authorization, model inference, retrieval, search, embedding, storage, feature flags, telemetry, and rate limiting. For each one, answer three questions. What is the user-visible result if it fails? What is the fallback? What is the downtime budget, expressed as a currency cap per incident? If you cannot answer all three, the dependency is not ready for production traffic.

An authentication dependency is usually the most dangerous because it can fail before your code runs. A model API can fail after you have already spent tokens. A search dependency can fail silently by returning stale, empty, or low-quality results. A storage dependency can fail by adding latency that makes the whole experience feel broken. Each one needs a different fallback. Do not treat them as interchangeable.

Set SLOs that match the user path

An SLO is a promise about a user path, not a service. Do not set an SLO for model availability if the user path is ask a question and get a cited answer. The SLO should cover the full path: authentication, retrieval, inference, response generation, and delivery. If any dependency fails, the SLO should tell you whether the user got a useful answer, a degraded answer, or no answer at all.

Use error budgets as the control mechanism. If a dependency consumes too much error budget, you stop adding features that depend on it. You add fallbacks. You add caching. You add circuit breakers. You add clearer product messaging. The budget is not a vanity metric. It is the point where engineering stops optimizing for the happy path and starts paying for the failure path.

A good SLO has three properties. It is observable from the user path. It is testable in a failure drill. It is tied to a cost decision. If the SLO is only a percentage on a dashboard, it is decoration. If it changes what you ship, what you cache, and what you tell the user, it is an operating control.

Define the fallback and cap the downtime cost

A fallback is a product decision, not just a technical one. For each dependency, choose the degraded mode in advance. If authentication is degraded, do you allow a limited read-only session? If search is degraded, do you serve cached answers with a freshness warning? If model inference is degraded, do you fall back to a smaller model, a rule-based response, or a queue? If storage is degraded, do you serve stale context or block the request? The fallback must be explicit enough that a user can tell the system is degraded without reading a status page.

Then cap the downtime cost. The downtime budget is a currency cap per incident: the maximum acceptable loss from a dependency failure before you change the architecture. If the budget is exceeded, the dependency is no longer a dependency. It is a single point of failure that must be removed, replicated, or replaced. The budget should be reviewed after every incident, not only after a major outage. If a dependency repeatedly consumes budget, the answer is not to raise the SLO. The answer is to reduce the dependency's power over the product.

Run failure drills that test the fallback, not just the alert. Kill the dependency. Simulate a misconfiguration. Return stale search results. Increase latency. Force authentication to fail for a subset of users. Then measure the user path. Did the fallback engage? Did the SLO reflect the degradation? Did the product message match the actual state? Did the user path stay inside the currency cap? If the drill only proves that an alert fired, it has not proven anything useful.

The checklist applies to the Microsoft issue's partial restoration state. Map the authentication dependency. Set an SLO on the multi-service user path. Define the fallback for continued search restoration. Cap the downtime cost. Run the drill. An AI product is only as reliable as its weakest serving dependency, and the cost of that weakness is not abstract. Mailbox connectivity was normal while search restoration continued, so the user path is still degraded; the fallback passes only if that degraded path keeps downtime_cost under downtime_budget, otherwise reduce or remove the search dependency.