On this page5 sections
Picture a team reviewing a new test after an outage. The test passes. Someone closes the follow-up ticket. In this hypothetical case, though, the failure needed two requests running at the same time. The new test politely waits for the first request to finish before starting the second. It never puts the system in the situation the team wanted to understand.
Asking “why?” can uncover that mismatch, provided each answer gives us something to investigate rather than something to put in the next box. The 5 Whys works backward from an event through contributing causes. It need not end at five, and it need not travel in a straight line. A useful explanation leads to a change whose effect we can examine, even when the result tells us our explanation was wrong.
Start with the requests that ran out of time
“The system was unreliable” leaves almost everything open. “Requests missed their deadlines” gives us something to follow. Where did they spend their time? In our hypothetical case, the investigation could begin with these proposed answers. None is a finding yet.
| Question | Possible explanation | What would help establish it? |
|---|---|---|
| Why did the requests miss their deadlines? | They spent too long waiting for the database. | Timing records for requests and database work during the affected interval. |
| Why did the database work take longer? | Too much expensive work was running at once. | Observations of query cost and how much work was active together. |
| Why could so much work run together? | The limit on simultaneous work was missing or did not work as intended. | The limit actually in effect, and behavior under a representative load. |
By the third question, there are already two promising paths. Each database query might have become more expensive. Or the database might be doing too many otherwise reasonable queries together. Both could be true. The investigation becomes more useful when it follows that fork instead of squeezing both explanations into one more “why.”
Let the second request start before the first finishes
Open the test from our first scene. Follow its setup, the requests it sends and the result it checks. Does the second request begin while the first is still running? If overlap is part of the proposed cause, the test needs to create that overlap and observe what follows. Now “insufficient testing” has become a specific gap someone can inspect.
The two database explanations suggest different changes. Making a query cheaper reduces work per request. Setting a concurrency limit, a cap on how many requests can run at once, controls simultaneous work. Either could help an overloaded database, but it acts on a different part of the problem. Testing one does not settle the other.
Repeated attempts can add another branch. Request traces can show a caller trying again after a timeout; configuration can show how many retries are allowed. A load test can explore whether those settings produce comparable traffic. Even then, slower queries and a surge of cache misses may still be competing explanations. A cache miss means the requested result was not available in the cache, so work has to continue elsewhere. The next observation needs to distinguish those possibilities, not simply make the existing story longer.
Suppose retries really are adding to demand. Reducing them under comparable conditions should change that demand. If it does not, another caller or a background job may deserve attention. That result can be frustrating when a fix was almost ready to declare finished. It is also how the investigation saves you from fixing the wrong thing.
Read diagram description
Illustrative hypothesis, not a confirmed incident: expensive queries and unconstrained concurrency can both contribute to overload. Timeouts and retries may then amplify demand. Test each proposed link against evidence. Diagram labels: Higher query cost: More work per request; Missing concurrency limit: More simultaneous work; Database slows: Requests take longer to complete; Deadlines expire: Clients may start retry attempts; Retries add demand: Additional attempts can worsen the slowdown.
Keep the delivered test and what it demonstrates as separate decisions. A closed ticket records an administrative state; inspect the delivered test to establish what was actually built. Then examine whether it exercises the relevant overlap and what happened when it did. If the review ends before those questions are answered, leave the questions open and explain how their answers could change the proposed fix.
Why did it also take forty minutes to respond?
Finding the overload mechanism does not account for every minute before someone reduced its impact. In a hypothetical forty-minute response, the alert might not name the affected operation. The database team might be busy with another incident. A rollback might need an approver who is unavailable. These are different obstacles, even if they all appear on the same timeline.
Google's blameless postmortem approach directs attention to the conditions in which people acted. Instead of stopping at “the engineer should have noticed,” ask what that person could see and do. A clearer alert, an escalation route with someone available and an accessible decision-maker solve different problems. The incident records need to establish which problem was present.
It would be just as easy to replace blame of one person with an unsupported story about the organization. Workload, incentives, documentation and chance may all matter. Describe the part the records support; leave the rest as questions rather than giving a larger group the same unsupported accusation.
Choose the next experiment, not the final box
Once you have a plausible mechanism, a proposed change and a way to evaluate it, the review has somewhere practical to pause. Deeper causes may remain. Keep the unresolved questions that could change the remedy, and weigh the likely value of investigating them against the work involved.
For the retry branch, compare a bounded retry policy, request deadlines and concurrency limits while the dependency is degraded. Adding jitter means varying when retries happen so they are spread across time. That may help, but it does not by itself set a maximum on simultaneous demand. Watch load, rejected requests and recovery time to learn which part of the overload the change actually affects.
Sometimes the next useful step is an agreement rather than a patch. Two teams may use “recovered” to mean different things. They need a shared measurement before they can judge the same change. Settling that question is progress, even if no code-change ticket comes out of the meeting.
Make room for the result you did not expect
A hypothetical finding can distinguish three things plainly: traces show repeated requests after timeouts; the team thinks retries amplified demand; a controlled comparison will test a bounded policy against the old one. The next reader can tell what was observed, what was inferred and what is still proposed. Put material unresolved exposure in the risk registry so it remains visible after the meeting.
Back at the new test, let those requests overlap and watch the behavior your explanation depends on. If the intervention removes the failure under the tested conditions, you have learned something about that change. If demand stays the same, another branch needs attention. Either way, the next decision comes from what the test showed, rather than from the reassuring fact that the worksheet has five answers.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- blameless postmortem approachsre.google