Opens in a new tab
Guide

Who can authorize the next incident mitigation?

In brief

When a rollback could affect migrated data, give the incident commander a way to reach the approver and verify what customers can do afterward.

5 min read

Is this the right starting point?
Before you read
Basic incident-response, deployment and data-migration concepts; use established service and data-risk authority paths.
What you will take away
Coordinate a disputed rollback by separating compatibility, approval, execution and recovery evidence through an accepted handoff.
Sources
A boat rests on shore with two ropes not yet connected to their posts, a metaphor for a rollback waiting on both data-compatibility evidence and an authorized decision.
On this page5 sections

In a hypothetical outage, Mira has two people waiting for her answer. One wants to roll back the deployment that preceded the errors. The other has just pointed out that the deployment migrated the database. Before anyone touches production, they need to find out whether the old code can use the data they have now, and who can authorize the change if it can.

Mira can get both questions moving. As incident commander, she can organize the investigation and set the next review point. Permission to direct the response does not automatically include permission to accept the consequences of a rollback that affects data.

Who can decide about this rollback?

Google’s incident response guidance lays out roles for coordinating work during an outage. The rollback tests what those roles let Mira decide. She needs to reach whoever can approve the proposed change within its scope, then know who will carry it out and who will check whether it helped. A single person can hold several of those responsibilities. The handoff should still make each one clear.

Prepare for the changes that fall outside a routine deployment. A schema rollback, a traffic move between regions or a destructive recovery step may need a different approver. List the actions each role can approve, who can cover for them and what would require the team to stop. In Mira’s case, that lets her bring the decision to someone who can make it instead of finding out during the outage who might have permission.

What the older code would have to do

The errors arrived after the deployment. That timing makes rollback a reasonable proposal and leaves the migration question open. Mira asks one responder to identify the rollback version, the data it would encounter and the evidence that would show it can work safely. Another responder assesses whether reducing traffic within a defined limit could ease the impact. Both can gather evidence while Mira coordinates any change to production.

Suppose the compatibility check is still open at the review point. A note that says “rollback ready” would hide the obstacle from the person deciding what to do next. Mira brings the unresolved check, the traffic option and their current impact estimates to the people authorized to decide. They can choose among the remaining options with the open risk in view.

A decision that can survive the shift change

By the time a new responder joins, the room may have moved on to traffic. The decision note gives them the part they missed. It says which version the team considered, that immediate rollback was deferred because compatibility was unresolved, who is checking it and when to revisit the choice. If someone approves a mitigation, add its authorized scope, the expected effect and the observation that would show it failed. Record execution and verification when each happens.

At the shift boundary, Mira briefs the next commander on the open question, its owner and the next review point. The incoming commander explicitly accepts coordination and the team hears who now leads, as Google’s incident management guidance recommends. Authority over the data change still follows the established approval path. To rehearse the handoff, use the incident decision log replay kit and try its branch where the execution result is missing. See what the incoming commander can actually conclude.

If a mitigation is applied, the next question is what customers can do. A successful command can confirm that an interface accepted the change while requests still fail. For a service that queues work, new requests can recover before the jobs from the outage are processed. Check the live service and the backlog separately. If old jobs need to run again, check whether replay can repeat a side effect before saying the earlier work is complete.

Service recovery and backlog recovery can diverge
Service recovery and backlog recovery can diverge. For asynchronous work, new requests may succeed while older jobs remain delayed. Report both states and check replay for duplicate side effects before claiming that all affected work has completed.
Check live requests and jobs from the outage separately. The first can recover while the backlog still needs attention.
Read diagram description

After applying a mitigation, check whether new work is succeeding and which older jobs remain delayed or failed. Report both results. If jobs need to run again, check for duplicate side effects and give the remaining work an owner.

Where the response lost twenty minutes

Suppose the post-incident review finds twenty minutes between the rollback proposal and a decision. Time spent checking the old code against current data means something different from time spent reaching someone who could approve the change. Split those waits from the time to get execution access, apply the action and check the outcome. Keep uncertain timestamps as estimates.

If the compatibility answer was ready and no approver was reachable, improve coverage or delegate a specific scope in advance. If two responders applied conflicting changes, make ownership of the production action visible. If the command succeeded and the team took too long to find out whether customers could use the service, improve the recovery check. Use a short note when one will do. The record earns its place by helping the next person decide or act.

Rehearse the decision with one disputed change

What to recordHypothetical rollbackWhat to fill in
Decision nowCan this version be rolled back after the migration?Name the decision in front of the team.
Evidence availableErrors followed the deployment. Compatibility is unknown.Write what you observed and what you still need to establish.
Proposed action and scopeRoll back one service version if compatibility is established.Name the target, affected data and prerequisites.
Deferred alternativeWait on immediate rollback while compatibility is unresolved.Record the reason someone else will need to understand.
Authority and fallbackUse the established route for service and data risk.Name the reachable approvers and who can cover.
Executor and verifierAssign both jobs before any change.Record who accepts each responsibility.
Stop or revisitStop the rollback plan if compatibility fails. Revisit when new evidence arrives or impact changes.Name the observation or time that brings the decision back.
OutcomeStill open until live requests and queued work have been checked.Record what recovered, what remains and who owns it.

Hand the filled worksheet to someone who missed the conversation. Ask what they would check first and whom they would call if the evidence came back favorable. Mira’s successor can name both without restarting the rollback debate. If your successor can name the test but has nobody to call, there is a preparation gap you can close before the next outage.

Put this into practice

Reconstruct one incident

Bring: Known timestamps, response notes and evidence of recovery. Leave unknown times blank.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next