Opens in a new tab
Guide

Let the urgent repair through without reopening every release

In brief

A paused pipeline can hold the change needed to restore service. An error-budget policy needs a precise route for that repair and a separate way to resume ordinary releases.

6 min read

Is this the right starting point?
Before you read
Practical release management and SLO-policy experience, including telemetry freshness, data compatibility and emergency authorization.
What you will take away
Design and rehearse release gates with explicit unknown states, artifact-scoped repair exceptions and evidence-based exit criteria.
Sources
Two brown parcels sit behind a closed purple gate, beside a small white arch containing an orange parcel marked with a wrench and a raised lever.
On this page4 sections

Two changes are waiting to deploy. One adds a feature. The other might repair the service that is failing right now. The release pipeline has paused them both.

That pause has a purpose: stop adding risk while reliability is poor. But the person coordinating the incident still needs a way to consider the repair without reopening the queue behind it. A release gate is the check that holds or admits a deployment. It is most useful when the team has agreed what it checks, which exceptions are possible and what will end the hold.

Consider a hypothetical repair that changes the database's schema, the structure of its stored data. An older application version is still serving requests. Whether it can read that new structure will matter more than how loudly someone asks for the repair.

How much allowance is left, and how quickly is it going?

A service's reliability target allows some amount of failure within a defined period. That allowance is its error budget. The remaining balance tells the release owner how much is unused within that window; it does not tell them everything about what is happening now.

For that, look at the pace of failure too. Burn rate compares the observed fraction of bad events with the fraction the target allows. A service can have budget left while failures are consuming it fast enough to need attention immediately.

A short burst and a continuing decline might have spent the same amount so far. Their next few hours could look quite different. A short measurement window notices changes quickly but can follow noise. A longer one smooths those fluctuations but may react late. Choose windows for the service's traffic and the consequence of waiting, then replay incidents to see which releases the gate would have held or missed.

Google's example policy connects reliability performance with release behavior. Its thresholds describe an example agreement. The team still needs to choose the agreement for this service, including how that waiting repair will be considered.

The quiet error chart has two possible explanations

While the repair waits, the gate asks for the latest errors and gets an empty result. Perhaps requests are succeeding. Perhaps the collector, which sends the measurements, stopped sending anything. The empty result alone cannot settle that question.

Give missing or out-of-date measurements a defined route: hold the release or request review, according to policy. Otherwise a monitoring failure can accidentally become permission to deploy.

Rehearse three separate inputs. First, provide fresh observations that show zero errors for a known set of requests and interval. Next, provide no series of measurements at all. Finally, provide an earlier healthy observation that is now too old to use. Only the first establishes the observed zero for that population and interval. The other two need the route for an unknown state.

The completeness lab supplies zero and missing cases. Test the expiry of a previously healthy observation using the actual collector and time window the gate relies on. A successful zero-error case alone would leave that different failure untested.

A release gate needs an explicit unknown state
A release gate needs an explicit unknown state. Illustrative policy structure, not prescribed thresholds. Use fresh evidence and a current change classification; recheck near execution. Any exception needs an exact scope, authorized approver and verification plan.
An illustrative release decision, with no prescribed thresholds. Fresh measurements and the kind of change determine the route. An allowed exception identifies the exact change, its authorized approver and how the result will be checked.
Follow the release through the decision

This conceptual diagram separates ordinary checks, a hold or review, and an exception permitted by policy. Recheck the measurements and change classification near execution; the drawing supplies no threshold or approval. SLO means service-level objective, the agreed reliability target. Diagram labels: Current release evidence: SLO, burn, incident state, freshness and change class; Known + within policy: Proceed through normal checks; Stale, missing or outside limits: Hold or request review per policy; Bounded exception, if permitted: Record exact change, approver, expiry and verifier.

A decision about this repair, with the older version still running

Return to the proposed schema repair. The incident lead wants its recovery benefit. The data owner can answer a more specific question: which stored-data structures work with which application versions, including the older version still in use?

If that version cannot read the new schema, the repair could replace the present fault with another. If compatibility checks pass for every version still serving, that particular obstacle has been removed. The expected benefit, other risks and how to verify recovery still belong in the decision. A compatibility result helps because it changes what is known, not because it makes the whole deployment risk-free.

The data owner should also identify what the change could invalidate and whether it can be reversed. The authorized risk owner weighs those findings against the benefit of restoring the service. Any combination that remains unresolved stays in the exception record. Calling the repair urgent does not answer the compatibility question.

If the emergency policy permits proceeding, bind the approval to the exact built artifact being deployed. Record its expected effect, responsible owner, verification plan and the point at which to reconsider or let the approval expire. That decision lets this repair move while the unrelated feature remains paused.

Just before execution, check that the artifact and the application versions are still the ones considered. A rebuilt repair or a newly discovered old instance changes the question. The same check belongs on direct deployment and emergency tools, not only the ordinary pipeline.

A commercial deadline may affect the tradeoff. Make clear who can accept it and who stops execution when a technical prerequisite fails. The incident lead then has a route to a decision without asking the data owner to turn an unknown into a reassuring answer.

Keeping every repair blocked can prolong the harm already happening. Preserve an emergency route with independent verification, and review how it was used afterward. It should help people make a bounded decision under pressure without pretending they can settle every risk during the incident.

When the feature at the back of the queue can move

The repair's exception does not decide when ordinary releases resume. Write down that separate exit: the required period of healthy operation, the repair work that must be complete and the approval needed to reopen releases.

If sustained deterioration caused the hold, one healthy sample may not be enough to end it. Choose the recovery observations and review time in advance. Different entry and exit criteria can keep the pipeline from repeatedly stopping and starting near a threshold, provided they fit the service and remain visible to the release owner.

Use the policy template to record those choices. Rehearse the repair beside the held feature, including missing measurements and an approval whose assumptions have gone stale. The repair should move only through its scoped exception. The feature waits for the recovery conditions that reopen ordinary delivery. Each change has a route forward, and nobody has to mistake permission for one for permission for both.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next