What did the postmortem fix prove?
Separate a completed task from evidence that the underlying risk was reduced.
Practical methods you can take back to your team.
Separate a completed task from evidence that the underlying risk was reduced.
Evidence can split a 5 Whys investigation into several contributing causes. A concurrency example shows how to turn those explanations into useful tests.
Conflicting restart instructions show where a source-backed notebook helps: finding the relevant passages, dates and exceptions for the runbook owner.
A reference design for duplicate deliveries and partial API success, with durable incident identifiers and summaries drawn from the resulting record.
Regional failures can disappear in fleet averages. Grafana queries, population labels and linked traces help responders find the affected requests.
A reusable prompt connects observations, competing explanations and the next read-only check, so an incomplete record can still support useful investigation.
Directory conventions orient the investigation. Mounts, namespaces, inodes and service permissions explain which storage the failing process actually encounters.
A concurrency change can improve throughput and worsen tail latency. Comparable workloads and process-level limits help explain whether the change is worth keeping.
An upload report becomes investigable through its operation, region and time window. That same detail connects the engineering result back to the customer.
A capacity warning needs enough lead time and a reachable responder who can act. The alert review separates useful urgency from a missing response arrangement.
Traffic assignment determines which behavior reaches a canary. Promotion also depends on the baseline, service objectives and state left behind by rollback.
Worked incident records explain the MTTD formula, uncertain onset times and the effect of including failures first reported by customers.