Opens in a new tab
Explainer

SRE KPIs: what to measure in a reliability scorecard

In brief

Build a scorecard that shows what users received, where response lost time, and how much work it took to keep the service running.

5 min read

Is this the right starting point?
Before you read
Basic familiarity with service reliability, incident response and on-call work.
What you will take away
Build an interpretable KPI scorecard that keeps user outcomes, response timing and operating effort separate.
Sources
A caliper sits against a wooden sphere beside a brass balance and an hourglass.
On this page5 sections

Imagine a quarter when incidents recover faster because the same engineer takes every difficult case. Customers get their service back sooner. The engineer’s planned reliability work keeps slipping. Both belong in the review, especially if the team is deciding whether it can carry the same workload next quarter.

A useful SRE scorecard keeps customer outcomes, response times and operating effort separate. Its key performance indicators, or KPIs, should help the team choose what to change: share the specialist’s knowledge or access, adjust staffing, or prevent the failure that keeps sending work their way.

Start with four questions for the scorecard

QuestionCandidate indicatorContext to retain
Are users receiving acceptable service?Service-level indicator (SLI) performance and service-level objective (SLO) attainmentEligible population, target, window, exclusions, and missing data.
How quickly is reliability allowance being used?Error-budget burn rateObservation period, request volume, and persistent versus brief changes.
Where does response lose time?Detection, engagement, mitigation, and recovery durationsTimestamp definitions, incident population, outliers, and unknown start times.
What does operating the service cost people?Actionable pages and active on-call effortNight interruptions, repeat pages, specialist support, and recovery time.

Leave the definitions beside the results. A request-success rate needs its eligible population and exclusions; a recovery time needs a clear endpoint. If you track deployment frequency or change failure rate, use them to explain the delivery context. More releases tell you how often the team shipped; the service indicators tell you what users received.

Keep three kinds of reliability evidence visible
Keep three measures separate: customer outcomes through SLIs and objectives; response performance through detection and recovery intervals; operating workload through interruptions, follow-up and specialist demand.
Keep three things visible side by side: what users received, where response lost time, and how much work it took to operate the service.
Read diagram description

Keep three things visible side by side: what users received, where response lost time, and how much work it took to operate the service. Diagram labels: Customer outcome: SLIs and service objectives; Response performance: Defined detection and recovery intervals; Operating workload: Interruptions, follow-up and specialist demand.

Check what changed behind a faster average

Suppose eight minor incidents take ten minutes each to recover and two major incidents take sixty minutes each. The mean is twenty minutes. In the next period, nine minor incidents and one major incident take those same respective times. The mean falls to fifteen minutes. Nobody got faster at handling either kind of incident; there was one fewer major case.

Show the count and duration distribution for each severity where there is enough data to compare. Decide what “MTTR” means in your review: time to repair, recover and resolve may stop at different points. For mean time to detect (MTTD), record how you know when impact started. Keep incidents with unknown starts visible as exclusions rather than quietly starting their clocks at the first page.

If one long incident seems at odds with the trend, use it to check who counted and whether a meaningful group was left out. Then compare the broader evidence before changing staffing or concluding the process got worse.

Record when the measurement changes

Reclassifying incidents can lower the count in a severity group. Suppressing an alert can lower page volume. Put the change in definition on the chart and keep evidence that can challenge the apparent improvement. Customer-discovered failures can reveal what a lower page count missed.

Give each row an owner and a decision it can inform. If customer-reported failures rise while pages fall, the service owner might review which failures escaped monitoring and return with findings at the next review. The feedback loop guide works through how to test what a change actually did.

Calculate request success, budget use and detection time

IndicatorCalculationExample and decision
Request success rateGood eligible requests ÷ all eligible requests × 100999,500 successful requests out of 1,000,000 gives 99.95%. Compare it with the agreed service target.
Error-budget consumptionBad eligible requests ÷ allowed bad requests × 100A 99.9% SLO allows 1,000 bad requests in that population. 500 bad requests consume 50% of the budget.
Error-budget burn rateObserved bad-request fraction ÷ allowed bad-request fractionA 0.5% error rate against a 0.1% allowance is 5× burn for that observation window.
Mean time to detectSum of measured detection durations ÷ incidents with valid measurements4, 32, and 11 minutes give 15.7 minutes. Investigate the 32-minute case and report excluded incidents.

Use the same eligible population and time window for the numerator and denominator in each calculation. Request errors and downtime minutes are different units. If timely completion is part of a good request, record the latency threshold alongside the result.

Use the guide to defining service level objectives to choose the outcome, then record the allowance and policy in the error budget worksheet. Google’s SLO implementation chapter explains how the indicator, objective and budget fit together.

Turn faster recovery into a staffing or engineering decision

Back in that faster quarter, compare incidents with the same definitions first. Then use the on-call workload record to count the help the page log missed: off-shift calls, follow-up work, and engineering time that the specialist gave up. You can then see whether recovery got easier for the team or depended more heavily on one person.

If the same diagnosis still needs that engineer, make time for others to practice it and get the access they need. If one defect keeps bringing the engineer back in, put prevention work on the plan. Temporary coverage can be the right call in a hard quarter. Before making it the staffing baseline, decide what happens to the work that coverage displaced.

Put this into practice

Inspect your detection delays

Bring: Incident impact-start and discovery times, the service and reporting window, and a record of incidents you cannot time reliably.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next