On this page5 sections
Someone suggests a rollback. Support asks for news. Three responders start inspecting workers. In this fictional Dispatch response, there is plenty of activity but no clear route from a discovery to a decision. The next useful change is to give the work owners.
The tabletop continues lesson 1: two exports from different accounts are verified late, the API health check passes and a worker deployment preceded the reports. Cause and full scope remain unknown. An illustrative training policy helps organize the response; the exercise needs neither production access nor external messages.
Let the impact choose the response
The verified delays already justify coordination. Severity now determines what that coordination should involve: who joins, how urgently they work and which escalation route applies. A number by itself cannot supply those answers; it needs an organization’s policy. Here are this exercise’s three local training levels:
| Training level | What is known | What happens next |
|---|---|---|
| Investigate | One report is unverified; scope is unknown. | An on-call responder checks the reported operation and records the result. |
| Coordinate | Multiple independently verified exports have missed their deadline. | Assign a response lead, maintain one shared record and involve the needed service owners. |
| Expand | Evidence shows broad loss of a critical operation, or a condition requiring specialist escalation. | Bring in the technical specialists and decision makers that impact requires. |
Two verified late exports put Dispatch at Coordinate. The healthy API check does not reverse that finding: it tests the API process, while the failed customer operation also depends on background work. We can organize a response without claiming every export is affected.
Now suppose an export reached the wrong user. Even one confirmed case needs the organization’s security or privacy escalation route. The nature of the harm can matter more than the count. Keep the impact statement current so the response follows the facts rather than an old label.
Three responsibilities, three accepted assignments
Someone must coordinate, someone must direct the technical work and someone must communicate. Google’s incident-management guidance separates these responsibilities so an absorbing investigation does not leave everything else unattended.
For Dispatch, Morgan accepts incident leadership, Riley takes operations and Casey takes communications. Morgan maintains priorities and check-ins. Riley directs technical investigation and proposed mitigations. Casey prepares updates from shared facts and brings support’s questions back to the team.
Morgan asks, “Riley, can you check whether the affected jobs are waiting or making progress, and report back at 14:15?” Riley accepts. That short exchange establishes a person, a result and a time to revisit it. Typing a name in a document would not establish that the person saw the request or could take it.
With only two responders, Morgan could handle leadership and communications while Riley handles operations. If updates keep interrupting coordination, ask someone to take communications and hand it over explicitly. The roles divide attention; unfilled boxes on a chart cannot do that.
Suppose Riley finds a useful log line just as support asks whether every customer is affected. Without Casey, Riley has to stop, reconstruct scope and answer. With a communications owner, Riley can record the finding once while Casey uses the shared facts to respond.
Morgan can then bring in a storage specialist, revise the next update or stop an unproductive investigation as new evidence arrives. Riley has room to pursue the technical question, and Casey has a current account to communicate.
A proposed rollback still needs the right authority
Riley proposes reverting the worker deployment because it happened before the delays. The timing makes it worth investigating, not a proven cause. This exercise’s change policy also requires service-owner approval and confirmation that the old worker can read data written by the new version.
Leading the incident does not automatically grant permission to change every production system or accept every risk. Riley’s deployment access does not establish compatibility. Morgan’s role does not replace the required service-owner decision.
Morgan can request compatibility evidence from Riley while bringing the service owner into the decision. Name who checks compatibility, who approves and who follows the affected exports through processing and timely new completions. Another passing API check would not establish that recovery.
Some familiar mitigations may already have authority delegated through a reviewed runbook. Use it where it applies; this is not a reason to add an approval queue to every routine action. This rollback needs explicit attention because its compatibility and approval requirements remain unresolved.
Give the next responder the current state
A responder joining now should not need the whole room to reconstruct the last ten minutes. A short working record can show where to begin:
Dispatch exports delayed | Investigating
Known impact: two verified late exports in two accounts
Full scope and cause: unknown
Training level: Coordinate
Morgan: incident lead, accepted
Riley: operations, accepted
Casey: communications, accepted
Riley: locate the wait for the affected jobs; report at 14:15 UTC
Riley: establish whether the previous worker can read current data
Service-owner approval: requested; awaiting acceptance
Proposed rollback: pending compatibility check and approval
Success check: affected jobs progress; new exports meet deadline
Next update: 14:15 UTCNotice the difference between “accepted” and “requested.” Morgan, Riley and Casey have taken their roles. The service-owner request is still outstanding. Morgan must follow it rather than assume the missing decision is already being handled.
Change the response when the facts change
Try the exercise with only two people. Assign technical, coordination and communication work, then write the request for more help. Consider two new observations separately: a third verified late export in another account, and a confirmed export exposed to the wrong user. What changes in each case?
See how the two-person response can adapt
Morgan accepts leadership and communications; Riley accepts operations. Morgan asks for a communications responder so updates do not obstruct coordination. A third verified delay adds an account to the impact statement and makes scope investigation more urgent, but alone does not establish broad loss of the critical operation. Confirmed wrong-user exposure immediately calls for the relevant security or privacy escalation. Neither finding approves the rollback. Riley’s compatibility evidence and the required service-owner decision must still be resolved, or another mitigation chosen through the applicable process.
Now a new fact has a place to go. Riley brings back evidence, Casey updates the people waiting and Morgan moves the rollback question through the right owners. The response can change direction without first stopping to rediscover who is doing what.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- incident-management guidancesre.google
