Turn a failure into an engineering decision

The ticket can say “retry bug fixed” while the next lost acknowledgement still produces two exports. A useful post-incident action has to connect the work being delivered to the condition being changed. That connection is what makes it possible to ask, later, whether the fix helped.

In this hypothetical Dispatch incident, workers retried timed-out storage writes with new operation identifiers. Some customers received duplicate exports and the backlog extended recovery. Stopping new retries contained the immediate problem. Now the engineering action needs to change what happens when the same uncertainty returns.

Explain the causal condition

“Storage was slow” describes a trigger, but not the whole mechanism. A timeout left the write outcome uncertain; a retry with a new identity created independent work. Together they explain the duplicate. Faster storage might make that sequence less common while leaving its retry semantics untouched.

A candidate repair preserves an idempotency key across retries and makes the storage-facing operation recognize completed work. Test it where the earlier logic failed: the write succeeds and its acknowledgement is lost. A normal successful request would never create that uncertainty, so it cannot establish whether the repair handles it.

Recreate the uncertainty that produced the duplicate

Start with one logical export, E-42. Allow its write to complete, then suppress the acknowledgement before the worker receives it. The worker sees a timeout. Storage has a completed result. That disagreement is the situation the repair has to understand.

Now let the worker retry. A new logical identity may make the receiver treat it as another export. With E-42 retained and a receiving path that remembers or discovers the existing outcome, it can return that result instead. The design needs to say where recognition happens and how long it remains valid. A field named “idempotency key” cannot perform that behavior by itself.

Look at the artifacts as well as the status messages. One green reply may hide two stored outputs; one stored output could still be the wrong report. The intended result is one correct logical export that the right user can retrieve. That is the service behavior the assertions are there to examine.

Separate delivery from effect

Specify the expected change and its verification together. The merge tells you an implementation was delivered. The lost-acknowledgement exercise tells you whether repeated attempts now produce one logical export under the tested conditions. Both facts belong in the action record.

Keep enough of the exercise to repeat it: starting state, lost response or injected delay, repeated identity, resulting artifacts, and expected count. Then follow the customer’s download path, not only the worker’s success log. A duplicate remains a defect when the API happens to show just one response.

Name the remaining exposure

Suppose the test covers only the newest export format. Older workers or a legacy storage path may still create duplicates. Record those populations, who owns them and what decision will close or accept that risk. A successful test must not quietly broaden into a claim about untested paths.

An alert for duplicate exports is a reasonable alternative to compare. It can make the defect easier to notice without preventing it. Both detection and prevention may be worth funding, but they change different moments in the failure. Describe the one the proposed work actually reaches.

Build a reviewable action card

Make the action record useful to the next engineer: name the owner, implementation scope, expected effect, repeatable test, completion evidence, and review date. In this exercise, the stated supported paths need both the idempotency change and the lost-response test. If some remain untested, retain that work explicitly instead of closing the action as universal prevention.

Ask a partner which future incident could still happen after the change. If the answer requires guessing the supported scope, clarify the action. A desk review without a partner is useful too; describe the review you actually performed.

Carry the result forward

Link the action to the service's reliability plan and relevant SLI so its effect can be revisited. Rare events may take too long to reappear for an incident count alone to verify prevention; a targeted fault test can supply earlier evidence without claiming perfect reliability.

The next engineer should be able to recreate the lost acknowledgement and see why it no longer creates another logical export. That is a useful thing to carry out of an incident review: an explanation that has become different behavior when the failure returns.

Sources

Postmortem culture discusses learning and follow-up. The incident, proposed implementation and test are fictional examples, not claims about a real production system.