Investigate the incident without undoing each other’s work

The worker specialist is ready to restart the machines. The storage specialist is looking at their open connections. Both want the reports to finish, but if the restart happens now, one person’s next step may erase the other person’s next clue.

That is the coordination problem in this fictional Dispatch exercise. Exports are late on old and new worker versions, and a storage timeout has been observed. The cause is still unknown. An incident lead and two specialists can cover more ground together, provided their questions meet in the same place and nobody changes that place without telling the others.

Bring back answers that belong together

“Check the workers” leaves a specialist with plenty to do and no obvious moment to stop. Try a narrower assignment: “At 14:20, report whether delayed jobs cluster by worker version in region A between 14:00 and 14:15, with counts and sample identifiers.” Compare successful and delayed jobs by version, region and size so that different kinds of work do not quietly become a verdict on different code.

The answer matters because it changes where the team looks next. If comparable jobs are mainly late on the new version, inspect that change. If both versions wait in much the same way, inspect what they share. The specialist still chooses how to query the data; the assignment makes clear which decision the result should help.

Meanwhile, ask the storage specialist for timeouts and response times in that same region and period. Otherwise the team could reconvene with a perfectly accurate account of delayed exports in region A and perfectly healthy storage in region B. Both people did the work. The two answers simply missed each other.

Agree on the observations each specialist will bring and when they will return. The lead follows customer impact and the connections between these investigations, rather than running every query personally. With fewer people, combine roles openly. If keeping everyone coordinated prevents a specialist from investigating, that is a reason to seek help, not an invisible obligation to do both at once.

Reading the restart procedure does not restart the worker

A person can inspect the current state, prepare a possible action or carry out an approved action. Name which of those the assignment allows. “Look into restarting” is uncomfortable wording when one reader hears research and another hears permission.

Before a change, name its target, required authority, expected effect and recovery check. Ask whose investigation it could disturb. The incident lead can coordinate that conversation without holding every production permission; the authority to act still needs to be established.

Can the restart wait? Make the tradeoff visible.

Now let the worker specialist propose a fleet-wide restart while the storage specialist captures connection evidence. The useful discussion is quite concrete: which current connections and waiting-job identifiers are needed, how long will capture take, and what are customers experiencing during those minutes?

If the impact allows a short wait, capture the evidence first, then decide whether a smaller restart is justified. The investigation keeps its view of the current state without assuming the whole fleet needs to change.

If waiting would prolong unacceptable harm, take that tradeoff to the authorized owner. Preserving evidence has value, but customers do not owe the investigation an unlimited wait. Record what the team knowingly loses by acting sooner, as well as why that loss is justified.

Once decided, put the instruction where both specialists can see it: “Capture current connection state first; worker specialist holds restart until the storage specialist reports or the lead revises this decision.” If urgency changes the plan, change that line too. A private agreement in one conversation is little comfort to the person still working under the old one.

Meet again around the delayed reports

At check-in, start with observations, then interpretations and what remains unanswered. Suppose storage reports a high p99: the response time at or below which 99% of its measured requests finished. Which requests, and during which window? Without those details, an alarming number may describe work other than the exports everyone is trying to explain.

Put the findings together, choose the next step and retire explanations the combined evidence no longer supports. More investigation is useful when someone can connect its answer to this picture. Adding another workstream without that connection may produce more updates and very little help for the waiting customer.

Finally, hand the workstream table and decision log to a replacement lead. They should be able to see who owns each question, which actions are allowed and what result would change the plan. The original lead can then leave without taking the investigation’s memory with them. The storage specialist can finish a check without wondering whether a restart is already on its way.

Sources

Google’s incident-response chapter describes coordinated work and response roles. The people, staffing and events here are fictional teaching material.