Opens in a new tab
Guide

Writing a post-incident review the next responder can use

In brief

A delayed rollback connects the incident timeline to the information responders had, the compatibility question they faced and the follow-up that remains.

5 min read

Is this the right starting point?
Before you read
Familiarity with incident participation, timelines and follow-up work.
What you will take away
Run a review that reconstructs available information and distinguishes a completed action item from verified recovery improvement.
Sources
An unfolded procedure leads to separate application and data inputs beneath a raised, unlatched checking arm.
On this page4 sections

“Make rollback faster” becomes an action item, the runbook gets an update, and the task closes. In this hypothetical review, the next responder can now find the command. They still do not know whether the old application version can read data already migrated by the new one. The task was completed; the difficult recovery decision remains.

A useful review makes that distinction visible. It explains what happened, reconstructs why the available choices looked the way they did, and leaves the next person enough context to continue the improvement. That requires more than a timeline and a list of tasks, but it does not require an elaborate meeting for every incident. A recurring, well-understood failure may warrant a focused correction; an unfamiliar failure spanning several services may need interviews and a fuller analysis.

The timeline needs the information available at the time

Start with a draft that participants can correct: customer impact, measurements, logs, change records, incident notes and actions taken. Identify time zones and estimated timestamps so the sequence is useful without implying a precision the evidence cannot support. Include people whose work happened outside the main incident channel, such as support staff or dependency owners. Their account may explain a delay that otherwise appears to be empty time.

The timestamp saying someone opened the runbook does not explain the wait. What could they learn from it? Did it describe data compatibility? Did they have permission to act? Which observation made waiting safer than rolling back? These questions connect the timeline to the choices people actually faced.

Give responders time to recover and a way to contribute asynchronously. Separate observations from interpretations, and leave conflicting accounts visible until evidence can resolve them. If participants disagree about when compatibility information became available, keep that disagreement in the draft. Settling it too early could make the rollback delay look avoidable for reasons the responders did not know at the time.

The defect and the delayed response have different causes

Explain both how the service failed and why restoring it took the time it did. The first account follows the trigger and the conditions that spread its effect; the second follows detection, routing, investigation, mitigation and recovery. Keeping both in view prevents a fix to the original defect from standing in for a fix to the response. An application change may remove the trigger while leaving the same permission gap for the next incident.

Look for the awkward steps that helped, too. A manual cross-check may have prevented an unsafe rollback. A call to a dependency owner may have supplied the compatibility detail. Removing them because they look slow could remove their protection. Explain why they were needed before deciding what should replace them.

The blameless-review guide develops this examination of decisions without making a person the explanation for the failure. The 5 Whys method can help explore a mechanism, provided a branching incident is allowed to retain its branches. In the rollback case, unclear instructions and uncertain data compatibility could both matter without being interchangeable causes.

What would the proposed fix let someone do?

In the rollback review, the follow-up has to answer whether the older application can use the migrated data. Updating the procedure is one piece of that work; checking compatibility is another. Naming who will perform the remaining check lets the review end with an accurate account of what is ready and what the next responder still cannot assume.

A compatibility exercise needs to identify the application versions and migrated data it covers. Its result can then be incorporated into the runbook: which recovery path worked, which did not, and which conditions were outside the exercise. The owner can judge the implementation and continuing maintenance cost against that specific improvement in the next responder’s choices.

A useful check of that follow-up is a replay with the compatibility result withheld. Can the next responder identify the unanswered question, the person responsible for resolving it and the action that remains unavailable? Then add the result and check whether the permitted next step changes. The decision-log replay kit provides a synthetic record for practicing this review; it does not establish that a real rollback is compatible.

Some reviews reveal useful understanding without identifying a worthwhile new control. In that case, preserve the explanation rather than manufacture action items to make the review look productive. Where material exposure does remain, distinguish unanswered questions and accepted risk from delivery work. A changed backup configuration, for example, cannot establish a successful restore that was never tested. The risk registry provides a home for consequential uncertainty that needs a later planning decision.

Carry the explanation into follow-up work
Carry the explanation into follow-up work
Proposed review flow. A corrected timeline supports analysis, and analysis supports a specific change. Completion of the task still needs a check of its intended effect.
Read diagram description

Proposed review flow. A corrected timeline supports analysis, and analysis supports a specific change. Completion of the task still needs a check of its intended effect. Diagram labels: Corrected record: Timeline and participants’ evidence; Explanation: Contributing conditions and difficult decisions; Selected follow-up: Owner, capacity and expected result; Later check: Did the intended behavior change?.

What another team needs from the review

Share the mechanism and the consequential choices with the people who can learn from them, using appropriate access controls and removing unnecessary sensitive detail. Google's postmortem analysis chapter gives further context for learning across incidents. The local record should make it possible to recognize a related condition without requiring another team to read the entire incident channel.

The next responder needs a usable answer: these application versions can read this migrated data under these conditions, while this remaining case still needs a check. Put the work needed to establish that answer into normal planning, with time to finish it. Then the runbook can change the recovery decision, not merely show that last incident’s action item was closed.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next