When the first mitigation helps but does not fix the incident

The request graph drops. The oldest report does not arrive.

In this fictional Dispatch exercise, the team has capped new submissions. Fewer exports are coming in, exactly as intended, but completions still trail arrivals. Some workers finish nothing, and storage timeouts continue. The cap has reduced demand without restoring processing.

Before calling the whole attempt a failure, keep those two results separate. The next step is to find out why work still cannot finish, while preserving whatever protection the lower arrival rate provides. Removing the cap in frustration could put more reports behind the same obstacle.

The first half of the idea worked

We can establish that the cap reached the intended region, accepted requests fell and the oldest work did not recover. Processing may remain blocked even with less new work arriving. That does not make the cap useless, and it does not tell us that restarting workers would repair storage.

The original hope had two steps: fewer arrivals would ease competition for resources, then workers would make progress. The request count confirms the first change in arrivals, not the hoped-for reduction in contention. Are workers spending less time waiting for storage or another resource? Are completions improving? Those observations help locate where the expected chain stopped.

Choose one delayed export and trace its last recorded step. Did a worker take it from the queue? Did a storage write begin, and did it return? Set those steps beside an export that finished. The first meaningful difference gives a specialist something smaller to investigate than “Dispatch is broken.”

The shared note can say, “Demand reduction is active; processing remains impaired.” Keep the cap's owner and removal condition beside that statement. The next responder needs to know both what changed and what must not be casually undone.

Give the specialist a question they can answer

A worker waiting on storage, one unable to connect and one stopping before the write begins need different investigations. Give the appropriate specialist the delayed and successful exports from the same region and interval, so the comparison does not quietly mix unrelated conditions.

A request such as “Need the storage owner to determine whether connections can recover without restarting clients, and approve the recovery action” names two needs: an explanation and an authorized decision. It is more useful than a general request for someone to look at storage.

The coordinator can bring people together and keep the investigation moving. They cannot supply another person's production permission. Name the next decision's owner and when to seek backup if that person remains unavailable.

Two changes would leave two new questions

Now someone proposes restarting workers and rolling back the release together. If customers recover, the combined mitigation may reduce harm while leaving us unsure which change helped. If they do not, we have two new conditions to untangle.

When time allows, prefer one bounded change with an expected result. Severe customer impact can justify combined mitigation; record why it was necessary and what remains uncertain. The point is not to protect a neat experiment at customers' expense.

Before repeating an earlier action, ask what relevant condition has changed. If nothing has, why should the same action work differently now? The answer may reveal an assumption worth revisiting.

For this exercise, narrow the question before making another change: is the worker blocked on an existing storage connection, or unable to create a new one? Neither answer promises a repair. Each gives the storage owner a particular state to explain.

Keep the useful part of the first attempt

The next update should state the customer impact, what the cap actually changed, the question being investigated and the next update time. “Mitigation failed” throws away the reduction in arrivals and can make the incoming responder start over.

Finish the exercise with that update, a revised action log and one justified next step. The customer who has waited longest is still waiting, so the work is not done. But the team now knows something it did not know before the cap. The handoff should carry that knowledge to the person who can authorize the next useful action.

Sources

Incident response covers coordination and escalation. Dispatch's events are fictional teaching material.