Opens in a new tab
Incident management · Lesson 3 of 11

The incident looks different from each screen

In brief

A missing report, a falling chart and a recent release seem to tell one story. Put their times and sources together before deciding which story it is.

4 min read

BeginnerLessonAbout reading levels
Is this the right starting point?
Before you read
No incident-response experience is required; use the supplied fictional evidence packet and a shared document.
What you will take away
Build a shared incident picture that separates observations from hypotheses, distinguishes event time from discovery time and assigns useful next checks.
Sources
Overlapping translucent blue, ivory and coral sheets show an arched stone bridge, with torn colored paper around their edges.
On this page5 sections

Imagine joining an incident call at 14:10. Someone is talking about a missing report. Someone else has a chart open. A third person says, “We deployed earlier.” You have been on the call for ten seconds, and already the deployment sounds guilty.

There may be a connection. First, though, put what each person knows on the same page. In this fictional Dispatch exercise, the team has four observations, arriving at different times. It is a tabletop discussion with invented evidence, not access to a failing service. All times are UTC on the same illustrative day.

EvidenceEvent timeTime learnedWhat it establishes
Customer says an export submitted at 14:01 is still missing.14:01 submission14:07One reported delayed operation.
Completion rate falls in a five-minute chart bucket.Bucket ending 14:0514:08A population-level change somewhere in that bucket.
Worker deployment completes.13:3514:09A change preceded discovery of impact.
Storage timeout appears in one worker log.14:0414:10One observed dependency error.

The report has two times attached

The customer submitted the export at 14:01 and told the team it was still missing at 14:07. If the incident record says only “14:07: exports stopped,” a responder joining later inherits a much stronger claim than anyone made. One late report has become a service-wide failure, with an apparently exact start time.

Keep the smaller, useful statement: “At 14:07 we learned that an export submitted at 14:01 was still missing.” Now someone can look for earlier and later completions around that job. Neither its submission time nor the time the message arrived establishes when a wider outage began.

The chart needs similar care. Its five-minute bucket gathers results across an interval and displays them together. A low completion rate in the bucket ending at 14:05 points you toward those five minutes. It does not identify 14:05 as the instant something broke. The table keeps both clocks visible: when something happened and when it reached the response.

Give the deployment a question, not a verdict

The deployment finished at 13:35. That makes it worth examining; it does not yet explain the missing report. Likewise, one storage timeout at 14:04 is an observed error, not proof that storage caused every delay. Keep those observations separate from the explanations people are trying out.

A useful next question is whether old workers are delaying exports too. Ask one owner to compare them with the new workers. Ask another to compare storage timeouts in the same region and period, and agree when both will report back. Those answers can distinguish a problem confined to new code from trouble in something both versions use. “Everyone have a look” is generous with effort and rather vague about what should come back.

Give actions their own place in the record. “Propose a rollback” and “rollback completed” describe very different systems for the next person to investigate. Record the proposed change, the authority required, its approval and, after execution, the observed result. The lead coordinates the response; that role does not automatically include permission to accept every production risk.

At 14:13, the old workers join the picture

Now add another invented observation: at 14:13, a responder finds delayed exports on the old worker version. The explanation “only the new version is broken” no longer fits. Leave it in the history with what was known when it was proposed, but change the current explanation and the next check.

It would be tempting to clear the deployment immediately. Not so fast. Both versions could be waiting on troubled storage, and the new version might have increased demand on that shared storage. The new observation narrows what the team can reasonably say; it does not settle every connection.

The shared record might now read: “Exports are delayed on both worker versions. We are checking whether storage waits account for the delay. The deployment preceded the reports, but its contribution is not established.” A responder can pick up that question without being asked to defend somebody else’s first guess.

A status update needs the affected operation and known scope alongside that current view, plus a time for the next update. People need to know what they cannot do and when they will hear more, not just which theory is getting attention on the call.

Leave a place for the next responder to begin

Imagine joining again, this time after the 14:13 finding. The document should let you find the customer impact, the source for each observation, the current explanations and the people checking them. Summarize the chat into that readable view, retaining links to the original evidence for anyone who needs the detail.

This exercise cannot reveal a real root cause; the evidence is invented. It can produce something a real response needs: a record that changes without hiding why. At 14:10, the recent deployment offered an easy story. At 14:13, the old workers have given everyone a more interesting question, and the next person through the door can see how the team got there.

Sources

Google’s incident-response chapter provides the coordination background. This article’s evidence packet and times are synthetic.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next