Measuring service success: did the customer get the report?

The customer returns to the download page. Five minutes ago, Dispatch accepted the request for a report. The page is still empty. Somewhere, a success counter may already have gone up because the website replied promptly. That counter and this customer are describing different experiences.

In this fictional exercise, Dispatch's promise starts when it accepts the export: the correct file should be downloadable within five minutes. A worksheet or spreadsheet is enough to count how often that happens. The five-minute deadline is teaching data, not a recommendation for every product.

Four requests, but only two reports on time

Write down four accepted exports: A, B, C and D. A and B deliver the correct file within five minutes. C takes seven. D has no recorded outcome, even after its five minutes have passed.

A and B kept the promise. C eventually produced a file, but too late for the result we are counting. D needs investigation. Perhaps its worker failed; perhaps the recording system lost the evidence. Neither possibility lets us mark D as a confirmed success yet.

That leaves two confirmed good exports out of four accepted ones: two divided by four, or 50%. This kind of measure is called a service-level indicator, or SLI. Here, the thing it tells us is quite concrete: how often the customer could download the right report on time.

Now imagine starting the worksheet from completed reports instead. D never gets a row. Two successes divided by the three visible outcomes becomes about 66.7%, without a single extra customer receiving a report. Start from accepted job IDs and attach whatever outcome you find to each one. D's blank space then remains a question to answer, rather than a way to improve the percentage.

Midnight does not move the deadline

D would look very different if you checked it only one minute after acceptance. It would still have four minutes left. A missing result cannot establish a missed deadline before that deadline arrives.

Put exports together according to when Dispatch accepted them. This group is a cohort. Before finalizing its result, let the last export in the group have the full five minutes. The acceptance time decides which group the request belongs to; the elapsed time decides whether its deadline can be judged.

Take the group accepted from 23:55 up to, but not including, midnight. Finalize it at 00:05. An export accepted at 23:59 and completed at 00:02 stays in that earlier group and meets the promise. The calendar changed while it was working. The customer's five-minute allowance did not.

Ten blank rows in a thousand

The same question survives a larger worksheet. Suppose every deadline has passed for this fictional group of 1,000 accepted exports:

What happened in the synthetic group	Exports	How they count
Correct file downloadable within five minutes	960	Good
Correct file downloadable after five minutes	20	Not good for this indicator
Processing is confirmed to have failed	10	Not good
No completion evidence after the deadline	10	Unresolved, not confirmed good

There are 960 confirmed good results out of 1,000 accepted exports, or 96%. The ten unresolved rows account for one percentage point. If later evidence establishes that all ten files were downloadable on time, the result rises to 97%. If none were, it stays at 96%. That 96% to 97% range comes from missing evidence; it is not a statistical confidence interval.

Deleting the ten blank rows would instead give 960 divided by 990, about 96.97%. The tidier worksheet would describe only the better-documented exports. Keeping all accepted exports in the bottom of the fraction, called the denominator, prevents a missing record from making the service look better.

One customer still needs one report

A worker might send the same completion message twice. A retry might carry the same operation ID. Neither creates a second customer's export or a second successful report. Another engineer needs to know how those duplicates are handled, along with the operation identifier, acceptance event, outcome evidence, deadline and acceptance interval, to reproduce your count.

Cancellation needs an explicit choice, too. In this exercise, cancelling after acceptance leaves the export in the count unless the written specification excludes it for a defensible product reason. Put that choice where both engineers can see it. Otherwise they can produce different answers while both believe they are counting successful exports.

The original acceptance record must also survive a worker crash. If a failed worker takes its row out of the starting list, the failure has removed itself from the calculation. Comparing accepted-operation counts with their eventual outcomes keeps the unfinished work findable.

There is one last place the report can fail its customer. The worker may create the correct file on time, but the download can be denied. That has not met this promise. If your records show only that the worker finished, name the measure “worker completion” and state that download access is unmeasured. It can still be a useful count, provided it is not mistaken for the customer's result.

The rows that keep the count honest

Keep four small cases beside the calculation: the 23:59 acceptance completed at 00:02, a repeated completion message, a file denied at download and an export only one minute old. The first can count as good in its original group. The duplicate counts once. The denied download is not good. The young export still has time left, so its result awaits judgment.

Those cases give a future query change something concrete to answer to. And the customer at the empty download page now has a place in the numbers: an accepted request that keeps its row, receives its full five-minute allowance and cannot disappear merely because no file arrived.

Sources

Google's guidance on implementing SLOs separates what an indicator is meant to measure from how it is implemented. Dispatch's counts and policy choices are illustrative teaching data.