Runbook example: checkout diagnosis and bounded mitigation

Adapt an illustrative procedure with stop conditions, rollback, escalation and independent recovery checks.

Illustrative dataset · Tool version 2 · Reviewed September 29, 2026

Could another responder follow the procedure?

Start with the checkout example, then replace each field with a service-specific instruction. Establish the access needed, the evidence that selects a branch, and the conditions that make a responder stop. The queue-worker variant provides a second starting point. Neither includes executable production commands.

Section	Illustrative starting text
Service and environment	Illustrative checkout API / staging
Owner and escalation	Service owner on call; escalate through the team’s approved incident route.
Required access	Confirm responder role and read-only monitoring access before the exercise.
Applies when / stop conditions	Checkout failures after a deployment. Stop if rollback compatibility or target ownership is uncertain.
Confirm impact	Check a synthetic checkout and eligible request failures in the same window.
Diagnosis and branches	Compare rollout timing, application pool waits and database pressure. Do not infer database saturation from pool occupancy alone.
Bounded mitigation	Use only the service-approved, bounded mitigation after checking preconditions. Record the target and expected effect.
Rollback and partial failure	Record the exact service-approved reversal. If partially applied, stop and establish current state before another action.
Recovery verification	Verify successful user outcomes independently; record the agreed observation window and unresolved impact.
Coordination and updates	Record incident owner, incident link and next update time.
Last exercise and review	Illustrative starting example only. Replace with service-specific instructions and exercise evidence.

What this example does not establish

Filled fields are not proof of a safe procedure. Record the last exercise and its evidence separately from the last edit.

Try a variation

Open the example, change one input, and review the result. Keep definitions and denominators constant when comparing scenarios. Save a named draft before changing to a different dataset.

These datasets and worksheets may be adapted for your team or training. Attribute AIOpsSRE and link to this example when redistributing. Third-party sources retain their own terms.

Get AIOpsSRE by email · Report a calculation or explanation issue