Opens in a new tab
Commentary

What a service owner needs to make recovery happen

In brief

Follow late invoices from a shared queue to the planning meeting. Find which approval, access and engineering commitments let the service owner recover them.

5 min read

Is this the right starting point?
Before you read
No formal SRE training required; familiarity with team responsibilities helps apply the hypothetical recovery example.
What you will take away
Distinguish doing, deciding, coordinating and verifying, and identify the specific access, authority or capacity gap blocking an owner.
Sources
Paper collage of three buildings connected to a blue database cylinder. Hands hold a transparent overlay, press a stamp and mark a calendar.
On this page4 sections

In a hypothetical planning meeting, a billing service owner brings a list of invoices customers received late. The service could accept new jobs again after the outage. The jobs that had stopped partway through were harder to recover. Some may already have written an invoice to the shared database, and only another team could replay the retained messages. The service owner answered for getting the invoices out. The team needed an agreement that let them do it.

Use that gap to test what service ownership means in practice. Follow one job from the queue to the customer's invoice. Then put the missing authority, access or engineering time into the plan.

Who can get the missing invoice to the customer?

Responsibility usually describes work someone does. Accountability usually means answering for an outcome or decision. Organizations use the words differently, so name the actual jobs. The billing owner can tell which invoices customers are missing. The database team can check whether a job already wrote its record. The queue operator can run an approved replay, which puts retained jobs back through processing. Someone still needs to agree on the batch and the conditions for stopping, and someone must verify what customers receive.

Coordination keeps those jobs aimed at recovery. Google’s incident response guidance explains how teams can assign and delegate distinct work. The coordinator does not gain the right to replay every customer's billing job by taking that role. Decide in advance who can authorize the proposed batch and who can cover that decision.

Doing, deciding and coordinating can belong to different people
Doing, deciding and coordinating can belong to different people
Name who will do, approve, coordinate and verify the work. In a rehearsal, check that each person has the access and coverage the role needs.
Read diagram description

The operator investigates or applies a change. The approver can accept the consequences of that change. The incident commander maintains priorities and handoffs. The verifier checks whether the affected user operation works. One person may fill more than one role.

The guide to SRE and platform engineering explains a related split. A platform team may provide the replay tool. The application team still needs to establish what running it would do to a job already partway through.

Follow one job that may have already written an invoice

Build a rehearsal with two synthetic records. One stopped before it wrote an invoice; the other wrote an invoice and stopped before it acknowledged completion. Ask the billing owner how they would distinguish those records, who could approve a batch of retries and who could run it. If replay can produce a second invoice, keep that case out of an ordinary batch until the team can reconcile it or show that retrying adds no second effect. The AWS guide to safe retries explains that property, called idempotency.

Try the route with the primary queue operator unavailable. If nobody can take up an approved batch, fix operator coverage. If the operator is ready and no one knows who can accept the risk of duplicate invoices, fix the approval boundary. If the team cannot match a queued job to a written invoice, there is evidence to improve before either approval or access will help. The drill can reveal those gaps without touching real invoices.

Give the repair a place in the plan

Suppose the rehearsal produces a workable process for a small batch, with manual checks against the database. That helps the next recovery. It leaves the owner with a process that may be too slow for a large backlog. The long-term change might be to give each invoice a stable key the billing service can use to recognize a job it already processed.

Make the capacity decision explicit if that work competes with a planned billing feature. If leadership defers it, record the remaining exposure, who made the decision and what will bring it back. A review date or a recovery drill that exceeds the time customers can wait gives the owner a concrete trigger. If billing and the queue team disagree over who can supply the change, bring the recovery evidence and the cost of waiting to the person who can settle the commitment.

Verify the agreement with the next rehearsal

“Improve replay” can fill a line in the plan without changing what the owner can do. Ask the teams to show that the retained job that never wrote an invoice produces one, and that retrying the already written job creates no duplicate. Name who will deliver the change and who will inspect the customer result. The follow-through guide develops that verification step. Keep the scope of the result and any remaining exposure in the risk registry.

When an incident tests the arrangement, use the engineering review to understand the decisions in the conditions people had at the time. Google’s guidance on postmortem culture explains why people need to be able to describe those conditions without fear of blame. Personnel or misconduct concerns belong in their own process.

At the next planning meeting, ask the billing owner to show what they can now promise for those late invoices. A reachable operator, a defined batch and a check for duplicates give that promise some substance. If the arrangement still depends on manual checks that cannot keep up, the team knows which engineering decision it has yet to make.

Source context

Source links appear within this article. Read them alongside the author’s analysis and evaluate the guidance against your environment.

Report an error or outdated detail

Keep reading

Where to go next