Choosing the reliability work that earns the next two weeks

There are two engineer-weeks left before the planning review and three proposals on the table. Fix the retries that create duplicate reports. Add workers to finish reports faster. Automate the stalled-job checks that keep pulling people away from other work.

All three sound useful. That is what makes the choice difficult.

In this fictional Dispatch exercise, the task is to turn that crowded table into a decision someone can fund. Bring the earlier service brief, customer-success measure, error-budget policy, on-call record and incident follow-up. A compact proposal, perhaps one page, should explain what the chosen work changes, why that matters and which useful work will wait.

Three proposals, three different kinds of relief

Candidate	What it is intended to change	Estimated effort	What we still need to know
Repair retry identity	Reduce duplicate work after a write's outcome becomes uncertain.	Two engineer-weeks	Which legacy paths the repair covers.
Add worker capacity	Increase throughput if workers are the limiting step.	One week plus operating cost	Whether storage remains the constraint.
Automate stalled-job triage	Reduce time spent on repetitive checks.	One week plus maintenance	Whether exceptions simply move to another queue.

These are teaching estimates, not measured costs or guaranteed benefits. They cannot support an exact financial ranking. They can help us notice that the proposals do not do the same job.

More workers may finish more reports when workers are the bottleneck. They cannot correct a retry that creates a second copy. Correct retries may remove wasted work, but they cannot promise enough capacity for a much larger customer load.

The automation proposal offers another kind of relief: fewer hours spent checking stuck jobs. Follow an exceptional case through the proposed tool, though. If it lands in a new queue for another team to inspect, some of the work has changed desks rather than disappeared.

What if extra workers repeat the same wasted work?

Suppose the recent missed deadlines cluster around storms of retries and overloaded storage. Repairing retry identity looks like a plausible first investment, alongside limits on retries and on how much new work the service admits. The idea is to stop repeated work from multiplying pressure on an already strained dependency. It is a working explanation for this scenario, not a benefit we have measured.

The identity has to change what the receiving path does. It must recognize an existing operation before repeating expensive work; adding a field alone does not prevent duplication. The lost-acknowledgement exercise follows that distinction. Limits on retries and new admissions also need real implemented behavior, not just a sentence in the proposal.

Extra workers remain an option. They help only if storage can serve the additional work happening at once. A short measurement or bounded experiment might settle that question before the team commits to another worker pool.

The stalled-job tool will have to wait

Spending both engineer-weeks on retry identity leaves no time for the stalled-job tool unless someone supplies more capacity. Say that directly. “Reliability matters” cannot choose between two improvements that both might help.

The proposal also needs people who can make the work possible: who authorizes the code and policy changes, which dependency team must participate and whose other work is displaced. Naming a service owner does not give that person access, time or another team's cooperation. A fundable proposal connects responsibility to an authorized piece of work with resources behind it.

Keep the customer in the discussion. Will more people receive correct reports on time? Will recovery take less time? Will responders regain hours for engineering? Those benefits need not collapse into one tidy number. Their mechanisms and costs should be clear enough for someone else to challenge.

Give the worker proposal its strongest case

Imagine measurements show the workers filling up before the slowdown while storage still has room to handle more. That could make added worker capacity the better first choice. Put that observation in the proposal now, while it can still change the decision.

For the retry repair, design a bounded load test that includes a lost acknowledgment. We want one logical export per operation and a stable queue under the stated load. A successful test would support the proposed mechanism in those conditions. Whether production customers actually benefit still needs representative measurements after rollout.

Make it possible to disagree with the plan

The final proposal needs the user promise and current shortfall, the evidence that matters, the proposed change and cost, the alternative being deferred and a way to evaluate the result. Include legacy paths or customer groups the test does not cover. Use as much space as this decision needs, not as much as a template offers.

Hand the proposal to a reviewer. Ask what they think is being funded, what it should change and which finding would reverse the choice. If their answer differs from yours, fix the reasoning before polishing the presentation.

At this point, we have a defended proposal, not an improved fictional service. An estimate is not a test; a delivered change is not yet an observed benefit. At the next review, the useful question will be whether the retry work helped customers receive reports, not simply whether the two weeks have gone by.

Sources

Implementing SLOs and SRE engagement discuss objectives and collaboration with service owners. The options and estimates here are synthetic planning material.