Error budgets: the same balance can call for a different decision

Two teams look at the same number: forty exports of remaining allowance. One repaired yesterday's fault and is watching reports arrive normally again. The other is watching new failures arrive. The dashboard balances match; the afternoons do not.

In this hypothetical Dispatch exercise, the measured window contains 100,000 eligible exports. The objective is for 99.9% to produce the correct, downloadable report within five minutes after acceptance. The allowance for falling short is called an error budget. It becomes useful when the team knows what its remaining amount, and the pace at which it is being used, should change about today's work.

Forty exports left in this measurement

At a 99.9% objective, the permitted shortfall is 0.1%. For our 100,000 exports, the calculation is 100,000 × (1 − 0.999) = 100. If 60 exports missed the indicator, they used 60% of that allowance, leaving 40 exports of margin.

That is a description of the window already measured, not forty failures set aside for whatever happens next. In a rolling window, new events enter while old ones leave. The population changes, so the allowance needs to be recalculated rather than carried forward like a fixed balance.

Now suppose the failure fraction is 1%. That is ten times the 0.1% implied by the objective. This ratio is the burn rate: how quickly the service is consuming allowance relative to the objective's permitted failure rate. If the same population, traffic pattern and rate continued, the allowance would last one tenth of the window. It is a conditional projection. The calculation cannot promise that traffic will cooperate with its assumptions.

What happens to today's release?

For this exercise, use a synthetic policy with a few clear consequences. Ordinary feature releases may proceed when the indicator is within target and recent burn is acceptable. Fast consumption calls for the service owner to assess and reduce risk. An exhausted budget pauses ordinary risk-increasing releases until recovery criteria are met. Incident repairs and urgent security work have a documented exception route.

What the team sees	What the policy asks it to do
Forty exports of current-window margin, with recent burn in the agreed acceptable range	Use normal release criteria and keep watching recent consumption.
Forty exports of current-window margin, with failures accelerating	Investigate the changing rate before treating the remaining allowance as comfortable.
The allowance is exhausted	Pause ordinary risk-increasing releases and identify the recovery work.
Telemetry disappears	Call the budget state unknown, use independent evidence and escalate the measurement failure.

Those decisions need people who can carry them out. Name who can pause a release, who can approve an exception and what allows ordinary work to resume. Without that, the next responder has a policy telling them to do something they may not have authority to do.

Yesterday's fault and this afternoon's failures

Return to our two teams. The first has a repaired fault behind it. Its normal release review should still consider any exposure left by yesterday's problem. The second has seen ten exports miss the objective in the last few minutes, and more are arriving. Its owner needs to find out whether a change or growing backlog is consuming reliability faster than the team can respond.

Recent burn gives the remaining allowance that sense of pace. Keep the observation window and the population beside the rate. A number calculated for one kind of traffic cannot quietly represent another, even if both appear under the same service name.

The way back to ordinary releases needs more than a better percentage, too. Old failures can age out of a rolling window while the cause waits to recur. Depending on the service, resuming work may require repairing that cause, seeing acceptable recent completions and reviewing the proposed release's risk. The policy should say which evidence makes the return reasonable.

A pause should still let the repair through

Suppose the budget is exhausted and a patch addresses the failures that are still occurring. Blocking that patch simply because releases are paused would work against the reason for the pause. The exception route needs to describe what the patch should improve, its deployment risk, who approves it, how the result will be checked and how it can be rolled back.

The label “reliability work” cannot supply those answers. Explain how the patch reduces the measured failure. Then an urgent repair and an ordinary feature request receive different treatment for a reason the next person can inspect, rather than because one arrived with a more persuasive label.

Can someone use it without the meeting?

A one-page policy should let a fresh reader find the indicator, objective, window, current allowance, recent burn and decision owners. As a proposed review exercise, give it to someone who was not part of the discussion and work through the four table cases plus the urgent patch. When they need an unwritten rule, the policy has found a missing explanation. This is an exercise to try, not a claim that readers have already tested this policy.

Keep severe individual failures visible alongside the aggregate. A small number of high-consequence failures can need an immediate incident response even when allowance remains. Forty exports of margin cannot tell you what happened to one particularly affected customer. The useful policy keeps that customer and operation beside the numbers, so the team can act while there is still something to protect.

Sources

Google's example error-budget policy and its chapter on alerting on SLOs explain policy and burn-rate concepts. Dispatch's numbers and policy here are a teaching example, not Google's policy or a universal standard.