On this page5 sections
Imagine a quarter when incidents recover faster because the same engineer takes every difficult case. Customers get their service back sooner. The engineer’s planned reliability work keeps slipping. Both belong in the review, especially if the team is deciding whether it can carry the same workload next quarter.
A useful SRE scorecard keeps customer outcomes, response times and operating effort separate. Its key performance indicators, or KPIs, should help the team choose what to change: share the specialist’s knowledge or access, adjust staffing, or prevent the failure that keeps sending work their way.
Start with four questions for the scorecard
| Question | Candidate indicator | Context to retain |
|---|---|---|
| Are users receiving acceptable service? | Service-level indicator (SLI) performance and service-level objective (SLO) attainment | Eligible population, target, window, exclusions, and missing data. |
| How quickly is reliability allowance being used? | Error-budget burn rate | Observation period, request volume, and persistent versus brief changes. |
| Where does response lose time? | Detection, engagement, mitigation, and recovery durations | Timestamp definitions, incident population, outliers, and unknown start times. |
| What does operating the service cost people? | Actionable pages and active on-call effort | Night interruptions, repeat pages, specialist support, and recovery time. |
Leave the definitions beside the results. A request-success rate needs its eligible population and exclusions; a recovery time needs a clear endpoint. If you track deployment frequency or change failure rate, use them to explain the delivery context. More releases tell you how often the team shipped; the service indicators tell you what users received.
Read diagram description
Keep three things visible side by side: what users received, where response lost time, and how much work it took to operate the service. Diagram labels: Customer outcome: SLIs and service objectives; Response performance: Defined detection and recovery intervals; Operating workload: Interruptions, follow-up and specialist demand.
Check what changed behind a faster average
Suppose eight minor incidents take ten minutes each to recover and two major incidents take sixty minutes each. The mean is twenty minutes. In the next period, nine minor incidents and one major incident take those same respective times. The mean falls to fifteen minutes. Nobody got faster at handling either kind of incident; there was one fewer major case.
Show the count and duration distribution for each severity where there is enough data to compare. Decide what “MTTR” means in your review: time to repair, recover and resolve may stop at different points. For mean time to detect (MTTD), record how you know when impact started. Keep incidents with unknown starts visible as exclusions rather than quietly starting their clocks at the first page.
If one long incident seems at odds with the trend, use it to check who counted and whether a meaningful group was left out. Then compare the broader evidence before changing staffing or concluding the process got worse.
Record when the measurement changes
Reclassifying incidents can lower the count in a severity group. Suppressing an alert can lower page volume. Put the change in definition on the chart and keep evidence that can challenge the apparent improvement. Customer-discovered failures can reveal what a lower page count missed.
Give each row an owner and a decision it can inform. If customer-reported failures rise while pages fall, the service owner might review which failures escaped monitoring and return with findings at the next review. The feedback loop guide works through how to test what a change actually did.
Calculate request success, budget use and detection time
| Indicator | Calculation | Example and decision |
|---|---|---|
| Request success rate | Good eligible requests ÷ all eligible requests × 100 | 999,500 successful requests out of 1,000,000 gives 99.95%. Compare it with the agreed service target. |
| Error-budget consumption | Bad eligible requests ÷ allowed bad requests × 100 | A 99.9% SLO allows 1,000 bad requests in that population. 500 bad requests consume 50% of the budget. |
| Error-budget burn rate | Observed bad-request fraction ÷ allowed bad-request fraction | A 0.5% error rate against a 0.1% allowance is 5× burn for that observation window. |
| Mean time to detect | Sum of measured detection durations ÷ incidents with valid measurements | 4, 32, and 11 minutes give 15.7 minutes. Investigate the 32-minute case and report excluded incidents. |
Use the same eligible population and time window for the numerator and denominator in each calculation. Request errors and downtime minutes are different units. If timely completion is part of a good request, record the latency threshold alongside the result.
Use the guide to defining service level objectives to choose the outcome, then record the allowance and policy in the error budget worksheet. Google’s SLO implementation chapter explains how the indicator, objective and budget fit together.
Turn faster recovery into a staffing or engineering decision
Back in that faster quarter, compare incidents with the same definitions first. Then use the on-call workload record to count the help the page log missed: off-shift calls, follow-up work, and engineering time that the specialist gave up. You can then see whether recovery got easier for the team or depended more heavily on one person.
If the same diagnosis still needs that engineer, make time for others to practice it and get the access they need. If one defect keeps bringing the engineer back in, put prevention work on the plan. Temporary coverage can be the right call in a hard quarter. Before making it the staffing baseline, decide what happens to the work that coverage displaced.
Put this into practice
Inspect your detection delays
Bring: Incident impact-start and discovery times, the service and reporting window, and a record of incidents you cannot time reliably.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- Google SRE: Implementing SLOssre.google
