After the incident: change what the next retry does

The report exists. The caller does not know that.

In this hypothetical Dispatch incident, a worker asks storage to write an export. Storage finishes the write, but its acknowledgment never reaches the worker. The worker is the caller at this boundary. After waiting long enough, it retries with a new operation ID, which the receiving path treats as new work. Some customers get duplicate reports, and the extra work makes the backlog take longer to clear.

Stopping new retries can contain the immediate problem. Preventing the next duplicate asks a different question: when a reply disappears, can the service recognize work it has already done? That is the behavior an engineering follow-up needs to change. “Retry bug fixed” is not quite enough to tell the next engineer how.

A faster reply would help. It would not remove the flaw.

Slow storage can set this incident in motion. Yet slowness alone does not create two exports. The timeout leaves the first write's outcome unknown; the new identity turns another attempt into an independent job. Faster storage might make that combination less common without making it impossible.

One possible repair is to carry the same identifier through every attempt at the same logical operation. That identifier is an idempotency key. The receiving path must use it to recognize completed work and return the existing outcome rather than do the work again.

To find out whether the repair works, we need the awkward moment back: the write succeeds, but its acknowledgment is lost. An ordinary successful request never puts the two sides in disagreement.

E-42 is finished on one side and uncertain on the other

Start the exercise with one logical export, E-42. Let the write finish, then suppress the acknowledgment before the worker receives it. Storage now holds a finished result. The worker sees a timeout. Let the worker retry.

A new identity may persuade the receiver to create another export. Keeping E-42 gives it a chance to recognize the first one, but only if the receiving path remembers or can discover that result. The design must say where that recognition happens and how long it remains valid. A field named “idempotency key” cannot remember anything by itself.

Look at the stored outputs, not just the reply. A green status could accompany two reports. A single report could contain the wrong result. What we want is one correct logical export, available to the right user.

Merged code and one correct report are different milestones

The merge tells us an implementation was delivered. The lost-reply exercise tells us whether repeated attempts produce one logical export under the conditions tested. Put both in the action record, with the expected behavior beside its verification. Neither needs to pretend it answers the other's question.

Leave enough detail for another engineer to repeat the experiment: the starting state, the lost response or injected delay, the identity used again, the resulting artifacts and their expected count. Then follow the customer's download path. A worker can truthfully announce success while the customer still cannot retrieve the report.

Which workers and storage paths remain untested?

Suppose the newest export format passes. An older worker or a different storage path may still create duplicates. Name those populations, who owns them and what would close or explicitly accept the remaining risk. The successful case cannot speak for a route the test never touched.

There may also be a good case for an alert that spots duplicate exports. That helps people discover the defect; it does not prevent it. Both changes could deserve funding. Explaining when each helps makes it easier to choose between them or justify doing both.

Leave the next engineer more than a closed ticket

A useful action record names the owner, supported scope, implementation, expected effect, repeatable test, completion evidence and review date. In this exercise, every path claimed as supported needs both the identity-preserving change and the lost-response test. Untested paths should remain visible, not disappear inside a claim of universal prevention.

Ask a partner which future incident could still happen after the repair. If they have to guess where it applies, the record needs another pass. A desk review can help too. Describe the review that actually happened; do not add an imaginary participant to make it sound more substantial.

Bring back the troublesome moment

Connect the action to the reliability plan and the measurement of the user-facing result, its service-level indicator. A rare failure may not happen again soon enough for an incident count to tell us whether prevention worked. The targeted fault test can give earlier evidence, though not a promise of perfect reliability.

Months later, another engineer should be able to let E-42 finish, lose its reply and watch the retry return the existing export. The postmortem has then done more than explain why there were two reports. It has left a way to show why, in that tested situation, there is now one.

Sources

Postmortem culture discusses learning and follow-up. Dispatch's incident, candidate repair and test are fictional examples, not a claim about a production system.