Opens in a new tab
Guide

Write a post-incident review the next responder can use

In brief

The team documented the rollback command. The next responder still needs to know if the old application can read the new data. A review should carry that question forward.

4 min read

Is this the right starting point?
Before you read
Basic incident-response, runbook and deployment-compatibility concepts; bring records participants can correct.
What you will take away
Turn a post-incident account into a specific follow-up whose effect on recovery decisions can be checked.
Sources
An unfolded procedure leads to separate application and data inputs beneath a raised, unlatched checking arm.
On this page4 sections

Imagine opening a post-incident review and finding that the team has already closed its rollback follow-up. The runbook now puts the old version’s deploy command at the top. There is still a question you would want answered before running it. Can that version read the data the new version already migrated?

A useful post-incident review leaves the next responder with a better answer than the team had during the incident. It records what made the failure possible, what slowed recovery and which of those conditions the follow-up can change. A familiar, well-understood failure may need a short account and one repair. When the outage crosses teams and services, it can take a fuller record to find out where each person’s view ended.

What the responder learned at 10:15

Build a draft people can correct. Bring the customer impact, measurements, change records, incident notes and actions together on one timeline. Mark time zones and estimates. The support engineer and the owner of a dependency might have learned something the responders in the main channel could only see later. Give people a chance to add that information asynchronously after they have had time to recover.

“Opened the runbook at 10:15” is a useful start. Underneath that line, say what the runbook offered and what the responder could do with it. In our example, it offered a command. The responder still needed permission to run it and an answer about what it would do to migrated data. If two accounts disagree about when the answer arrived, keep the disagreement visible. The recovery choice looks quite different on either side of that moment.

The defect and the wait tell different stories

Follow the failure from the first trigger through the conditions that let it spread. Then follow the people who had to detect it, get the right teams together and restore the service. A patch for the original defect might do nothing for the responder who still has to find someone with rollback access.

Some of that wait may have protected customers. A dependency owner might have recognized that restoring the old application would leave it reading a new data format. A manual check that looks slow in the timeline deserves a closer look before anyone removes it. The blameless review guide helps reconstruct a decision in context, and 5 Whys gives the team a way to explore the contributing branches. Runbook clarity and data compatibility belong on different branches here.

Try the rollback the runbook describes

The documentation owner can deliver a clearer command today. Give the compatibility exercise its own owner and enough time to run it safely. Say which application versions and migrated data it will cover. Record the recovery path that worked under those conditions, the paths that failed and the cases the team did not exercise. That gives the owner enough information to weigh the remaining work and the cost of keeping another recovery path available.

There is an inexpensive rehearsal the team can do before the technical exercise. Give another responder the timeline with the compatibility result withheld. Can they name the missing fact, find its owner and recognize which action has to wait? Add a representative result and see whether it changes the permitted action. The decision-log replay kit offers synthetic material for that kind of practice. An actual compatibility test still has to supply the answer for the service.

Sometimes the rehearsal shows why the wait was right. Preserve that finding without manufacturing another task. If compatibility remains untested after the runbook edit, record the edit as delivered and keep the data question open. If an authorized owner accepts the remaining exposure, record that decision separately. A risk registry can keep a consequential unresolved question visible between the review and the next planning decision.

Carry the explanation into follow-up work
A corrected timeline leads to an explanation, an assigned follow-up and a check of whether behavior changed.
The corrected timeline explains the compatibility question. Give the follow-up an owner and check whether it changes the recovery choice.
Read diagram description

The sequence starts with a timeline corrected by participants. It moves through contributing conditions and difficult decisions to a selected follow-up with time, an owner and an expected result. The last step asks whether the intended behavior changed.

What to carry into the next incident

Share the mechanism and the consequential choices with the teams that can use them. Remove sensitive details those teams do not need and set access accordingly. Google’s SRE Workbook analysis of past postmortems illustrates how a consistent record makes it possible to look across incidents for recurring triggers and contributing conditions. A team reading about this one should be able to spot the rollback question in its own service without paging through the original incident chat.

The next time someone opens the runbook, they should see which versions have been tested against which migrated data and where the evidence stops. Put the work needed to obtain that answer into the plan. The command at the top is easier to find now. The answer beside it is what lets someone decide whether to run it.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next