Fewer requests arrived. Did the mitigation work?

On the incident call, one person wants to roll back the workers. Another wants to accept fewer export requests. Customers are still waiting for their reports, so either suggestion sounds better than watching the queue grow. But the two changes would help in different ways. One might do exactly what was requested without getting a single report closer to completion.

In this fictional Dispatch exercise, storage writes time out, workers retry and queued exports get older. A new worker version arrived earlier, but old workers are delayed too. A change meant to reduce that impact is a mitigation. Choosing one starts with following what it would do to the unfinished work, rather than choosing the command everyone knows best.

One option changes arrivals. The other changes workers.

Picture a new request entering the queue. A worker takes it, writes the report to storage and finishes the export. A submission limit admits fewer new requests, giving workers a chance to catch up if they can still finish reports. If nothing can finish, fewer arrivals merely slow the growth of the queue.

A rollback replaces the worker program. It could help if the newer version increased demand on shared storage; old workers can suffer from that extra demand too. Before choosing it, though, find out whether the old program can read data the new version has already written. The files do not return to their old format just because the executable does.

Option	Evidence needed	Important risk	Verification
Roll back workers	New behavior contributes to demand; old version remains compatible.	Old binary cannot read newly written data.	Comparable exports improve; no new compatibility failures.
Limit submissions	Incoming work exceeds useful completion capacity.	New customers cannot start work; unsafe retries add load.	Queue age trends down; rejection behavior is clear and bounded.

Read across either row and the tradeoff becomes visible. The rollback needs a connection between new behavior and demand, plus compatible data. The limit needs incoming work to exceed useful completion capacity, and it turns away customers who want to begin. Neither option is free simply because it is familiar.

A choice on the call still needs an owner

The incident lead keeps this decision aligned with the rest of the response. Authority to change production follows the organization’s policy: the service owner authorizes changes, involving dependency or data owners where needed. Leading the conversation does not automatically include the right to accept every customer or data risk.

In the tabletop record, name the approver, exact scope, start time, stopping condition and recovery check before treating an action as chosen. Record execution separately. The next responder needs to know whether the team agreed on a cap or actually applied it; those are different explanations for what appears on the chart.

Can five spare completions a second clear the queue?

Suppose compatibility for a rollback is still unknown, and the service owner has approved a limited submission cap in region A. Its intended benefit is straightforward: fewer exports arrive, leaving enough completion capacity to work through the waiting reports.

Use invented teaching numbers. Before the cap, 30 exports arrive each second and 20 finish. Ten more join the waiting line every second. After the cap, suppose 15 arrive while 20 still finish. Now five more leave than enter each second. With 300 waiting, that spare capacity would clear the queue in about 60 seconds: 300 divided by five.

That small calculation assumes the rates stay steady. It is not a forecast for Dispatch. If completions fall to zero, there is no five-per-second recovery to spend. Compare arrivals, completions, queue count, oldest queued age and the treatment of new requests before and after the action, using the same region and comparable time windows. Together they show whether the limit took effect and whether accepted work is moving.

The cap worked. The reports are still late.

Now introduce a less satisfying fictional result: fewer requests are accepted, but completions remain below arrivals. The queue grows and the oldest waiting export gets older. The cap has changed the arrivals as intended. It has not restored enough processing to catch up.

The completion rate and growing queue establish that shortfall. Oldest age by itself would not: one old job can keep aging for a while even as other jobs drain. Follow actual older jobs as well as the graph, and check whether workers can make progress. A falling API error rate tells you something about those requests; it does not show that the missing reports reached their customers.

Also watch what rejected clients do. If they retry aggressively, they can add the very load the cap was meant to remove. When that behavior makes the mitigation harmful, pause and revise the approach with the owner. Otherwise, keep justified demand reduction while investigating the processing problem that remains.

Leave the next responder the alternatives, the reason for choosing the cap, the approval and the measured change. “Arrivals fell, but the backlog still grew” may be less satisfying to announce than “fixed.” It is much more useful to the person taking over. They can see what the action bought and why customers are still waiting for the part that matters to them.

Sources

Google’s incident-response chapter provides coordination context. The option matrix, rates and outcomes here are hypothetical, not production instructions for an actual service.