On this page4 sections
Imagine opening a post-incident review and finding that the team has already closed its rollback follow-up. The runbook now puts the old version’s deploy command at the top. There is still a question you would want answered before running it. Can that version read the data the new version already migrated?
A useful post-incident review leaves the next responder with a better answer than the team had during the incident. It records what made the failure possible, what slowed recovery and which of those conditions the follow-up can change. A familiar, well-understood failure may need a short account and one repair. When the outage crosses teams and services, it can take a fuller record to find out where each person’s view ended.
What the responder learned at 10:15
Build a draft people can correct. Bring the customer impact, measurements, change records, incident notes and actions together on one timeline. Mark time zones and estimates. The support engineer and the owner of a dependency might have learned something the responders in the main channel could only see later. Give people a chance to add that information asynchronously after they have had time to recover.
“Opened the runbook at 10:15” is a useful start. Underneath that line, say what the runbook offered and what the responder could do with it. In our example, it offered a command. The responder still needed permission to run it and an answer about what it would do to migrated data. If two accounts disagree about when the answer arrived, keep the disagreement visible. The recovery choice looks quite different on either side of that moment.
The defect and the wait tell different stories
Follow the failure from the first trigger through the conditions that let it spread. Then follow the people who had to detect it, get the right teams together and restore the service. A patch for the original defect might do nothing for the responder who still has to find someone with rollback access.
Some of that wait may have protected customers. A dependency owner might have recognized that restoring the old application would leave it reading a new data format. A manual check that looks slow in the timeline deserves a closer look before anyone removes it. The blameless review guide helps reconstruct a decision in context, and 5 Whys gives the team a way to explore the contributing branches. Runbook clarity and data compatibility belong on different branches here.
Try the rollback the runbook describes
The documentation owner can deliver a clearer command today. Give the compatibility exercise its own owner and enough time to run it safely. Say which application versions and migrated data it will cover. Record the recovery path that worked under those conditions, the paths that failed and the cases the team did not exercise. That gives the owner enough information to weigh the remaining work and the cost of keeping another recovery path available.
There is an inexpensive rehearsal the team can do before the technical exercise. Give another responder the timeline with the compatibility result withheld. Can they name the missing fact, find its owner and recognize which action has to wait? Add a representative result and see whether it changes the permitted action. The decision-log replay kit offers synthetic material for that kind of practice. An actual compatibility test still has to supply the answer for the service.
Sometimes the rehearsal shows why the wait was right. Preserve that finding without manufacturing another task. If compatibility remains untested after the runbook edit, record the edit as delivered and keep the data question open. If an authorized owner accepts the remaining exposure, record that decision separately. A risk registry can keep a consequential unresolved question visible between the review and the next planning decision.
Read diagram description
The sequence starts with a timeline corrected by participants. It moves through contributing conditions and difficult decisions to a selected follow-up with time, an owner and an expected result. The last step asks whether the intended behavior changed.
What to carry into the next incident
Share the mechanism and the consequential choices with the teams that can use them. Remove sensitive details those teams do not need and set access accordingly. Google’s SRE Workbook analysis of past postmortems illustrates how a consistent record makes it possible to look across incidents for recurring triggers and contributing conditions. A team reading about this one should be able to spot the rollback question in its own service without paging through the original incident chat.
The next time someone opens the runbook, they should see which versions have been tested against which migrated data and where the evidence stops. Put the work needed to obtain that answer into the plan. The command at the top is easier to find now. The answer beside it is what lets someone decide whether to run it.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- postmortem analysis chaptersre.google
