Opens in a new tab

3 steps · Incident response

Review an incident

Reconstruct the response, review corrective actions and update the runbook.

BringIncident timestamps, response notes, action owners and recovery evidence.

Finish withA timing report, a follow-up action register and a revised runbook draft.

  1. Step 1

    Reconstruct the response

    The timeline separates discovery, investigation, mitigation and recovery so a long incident becomes a sequence you can inspect. Enter the times supported by the record, with observation bounds when onset is uncertain.

    Before moving on: A long interval should point to the notes or events that explain what responders were doing during it. Keep unsuccessful mitigation attempts in the record.

    Before leaving, use Save file draft to keep the editable timeline and retain the full exported report. Prepare next draft carries only incident notes and a timing summary into the action tracker, not the milestone timestamps, evidence or mitigation-attempt rows. Open and review that summary in the same browser tab.

  2. Step 2

    Review the follow-up actions

    Take a delay or failed mitigation from the timeline and describe the problem the proposed action should change. Then name a test that could show the difference. This keeps an action such as “improve the runbook” connected to the response step that actually stalled.

    Before moving on: Delivered means the change exists. Verified means the relevant check was run and its result supports the intended improvement. Record any part of the original failure scenario that remains unresolved.

    Keep the action register, then select an action and use Prepare next draft. The runbook receives its scenario, proposed change, owner, test and evidence, but not its due date, status or remaining exposure. Review the transferred fields and carry relevant unresolved exposure into the runbook yourself. This transfer does not create an exercise sign-off.

  3. Step 3

    Update and exercise the runbook

    When an action changes how the service is diagnosed or recovered, add that procedure to the runbook. Include the symptom that selects it, the action’s limits and the observable recovery result.

    Before moving on: During an exercise, another responder should be able to follow the applicable branch and identify recovery. Record any missing access or unclear instruction with the exercise result.

    Export the runbook for team review. The draft does not validate the procedure for your service.

Original combined worksheet and v1 drafts

Your previous toolkit files still work here. Focused tools also accept the relevant fields from older drafts.

Keep your working draft

Entries stay on this page. Download a draft to reopen later; nothing is saved automatically or sent to us.

This draft contains the timeline, incident notes, Markdown runbook and error-budget entries. Other tools have their own drafts.

01 / Reconstruct the response

Incident timeline calculator

Enter dates and times in UTC, including the date when an incident crosses midnight. Leave uncertain timestamps blank.

Detection
Unknown
Acknowledgment
Unknown
Detection to investigation
Unknown
Investigation to recovery
Unknown
Mitigation to recovery
Unknown
Total impact
Unknown

Intervals use the named endpoints. This tool assumes one response sequence; for repeated mitigations, use the worksheet to record each attempt. A single incident supplies durations, not MTTD or MTTR averages. Unknown onset means unknown detection duration, not zero.

02 / Preserve the evidence

Editable incident worksheet

Add impact, evidence, recovery checks, and follow-up. The download includes your current timeline and calculated intervals.

Read the postmortem follow-up guide
03 / Prepare the next response

Editable SRE runbook

Fill in this existing AIOpsSRE template for one service and failure mode. Exercise the procedure before using it during an incident.

Download blank templateRead the runbook guide
04 / Check the allowance

Request-based error-budget calculator

Use eligible and bad events from the same measurement window. A bad event fails your defined success or latency condition. This is an event allowance, not a downtime allowance.

Illustrative starting values: 99.9% objective, 1,000,000 eligible events, 250 bad events.

Allowed bad events
1,000
Remaining allowance
750
Budget consumed
25%
Good-event rate
99.975%
Allowance remaining

750 bad events remain in this measurement window.

25%

Use your agreed release policy to decide the next action. This indicator describes consumption, not a release recommendation.

Allowed = eligible × (1 − target / 100). Remaining = allowed − bad. Negative remaining means the allowance is exceeded. Fractional allowances are shown without rounding to whole events; display values are rounded to three decimals. Missing telemetry can make these inputs incomplete. This calculation does not authorize or block a release.

Calculate burn rateRead the error-budget guide
# Incident worksheet

All timestamps are UTC. Unknown times are left unknown.

Impact began: Unknown
Incident detected: Unknown
Page acknowledged: Unknown
Investigation began: Unknown
Mitigation applied: Unknown
Service recovered: Unknown

## Timing
Detection: Unknown
Acknowledgment: Unknown
Detection to investigation: Unknown
Investigation to recovery: Unknown
Mitigation to recovery: Unknown
Total impact: Unknown

Single-incident durations are not mean metrics.

## Incident context
Service / environment:
Incident owner and record:
User impact and affected population:
Discovery method (monitoring / engineer / customer):
Timestamp evidence, proxies and uncertainty:

## Response and recovery
Actions taken and observed results:
Customer-outcome check and observation window:
Recovery evidence / unresolved impact:

## Follow-up
Failure scenario:
Proposed treatment:
Test that would demonstrate the intended improvement:
Remaining exposure:
Owner / due date / evidence link:


# Runbook: [specific failure mode]

Status: Draft, complete and exercise before operational use.
Service / environment:
Owner and escalation route:
Related alert:
Required access:
Last edited:
Last exercised / evidence:

## Scope and stop conditions
Applies when:
Does not apply when:
Stop and escalate if:

## Confirm impact
Customer symptom:
Exact dashboard / query and time window:
Expected healthy result:
Observed failure result:
Data freshness check:

## Initial coordination
Acknowledge via:
Incident record / channel:
Response lead and communication owner:
Next update time:

## Diagnosis
1. Check [specific condition] using [read-only query or procedure].
   - If [result], continue to [step].
   - If [other result], escalate to [reachable owner].
2. Check recent changes and relevant dependencies.
   - Evidence supporting current hypothesis:
   - Evidence contradicting it:

## Bounded mitigation
Action and approved workflow:
Required preconditions:
Explicit target / environment:
Maximum scope and attempts:
Authorization required:
Expected effect and time limit:
Partial-failure handling:
Rollback or recovery procedure, if supported:
Stop condition:

## Verify recovery
Independent customer-outcome check:
Dependency / resource check:
Observation window:
Result: verified / unsuccessful / inconclusive
If unsuccessful or inconclusive, escalate via:
Evidence to hand over: timeline, impact, checks, actions, results, uncertainties.

## Follow-through
Record recovery time and remaining risks.
Assign an owner and due date to unresolved work.
Correct procedure gaps found during use.
Schedule an appropriate review and repeat exercise.


# Error-budget worksheet

Illustrative example
Service and window: [Not specified]
SLO: 99.9%
Eligible: 1000000
Bad: 250
Allowed: 1000
Remaining: 750
Consumed: 25%
Event allowance is not downtime allowance.

Understand the measures behind the tools

Detection and MTTD

Detection time starts when impact begins and ends when someone discovers it. MTTD averages those intervals across a defined set of incidents. Keep unknown start times visible: leaving the slowest, hardest-to-reconstruct incidents out can make the average look better without improving detection.

Reviewed 2026-09-29. Guide-to-tool alignment reviewed; v2 arithmetic, imports and explicit assumptions checked with illustrative fixtures.

Incident response

An alert, an investigation and a recovered service happen at different times. Record those stages separately, then use the notes to explain what happened between them. A completed rollback is an action; recovery still needs evidence that the affected user operation works again.

Reviewed 2026-09-29. Guide-to-tool alignment reviewed; v2 arithmetic, imports and explicit assumptions checked with illustrative fixtures.

Runbooks

A runbook helps the next responder move from a recognizable symptom to a check, an appropriate action and a way to tell whether it worked. Fill it for one service, then rehearse it with someone who did not write it. Their first unanswered question shows where the procedure needs more explanation.

Reviewed 2026-09-29. Guide-to-tool alignment reviewed; v2 arithmetic, imports and explicit assumptions checked with illustrative fixtures.

Postmortem follow-through

A closed corrective-action ticket records a status, not proof that the change was delivered. Confirm what actually changed, then use a repeatable test to see whether it changes the failure that mattered. Keep the original scenario, the change, its test and any remaining exposure together, so the next review can distinguish a finished task from a demonstrated improvement.

Reviewed 2026-09-29. Guide-to-tool alignment reviewed; v2 arithmetic, imports and explicit assumptions checked with illustrative fixtures.

Error budgets

An error budget is the amount of unsuccessful service your SLO permits within its measurement window. For a request-based SLO, multiply eligible requests by the permitted failure fraction. At 99.9%, one million requests allow 1,000 bad requests; those units cannot be converted directly into downtime. Your designated decision owner applies the agreed service policy, including any exception, to the remaining allowance; the calculation itself does not authorize a release.

Reviewed 2026-09-29. Guide-to-tool alignment reviewed; v2 arithmetic, imports and explicit assumptions checked with illustrative fixtures.

SRE KPIs

A useful KPI helps answer a service question. For a timing metric, define when its clock starts and stops. Identify which incidents or requests count and which observations are missing. Inspect individual cases before trusting an average: a shorter response time means little if the measurement stops before users recover.

Reviewed 2026-09-29. Guide-to-tool alignment reviewed; v2 arithmetic, imports and explicit assumptions checked with illustrative fixtures.

Sources and calculation scope

The timeline follows the explicit endpoints in our linked metric guides. The worked example is illustrative. Calculator checks verify arithmetic and input handling; they do not establish production outcomes.

For future guides and resources, get AIOpsSRE by email or follow RSS.