Prometheus latency SLOs: counting the searches that actually helped

Imagine asking for the runbook during an incident and getting an error in ten milliseconds. The service was quick. You are still missing the runbook. A useful measurement has to count answers that succeed and arrive on time, while keeping failed searches among the requests it was supposed to serve.

Prometheus can keep counts at chosen duration boundaries rather than only a collection of individual stopwatch readings. Those counts, stored in a histogram, let us calculate the fraction that met the promise. That fraction is a service-level indicator, or SLI; the target it should meet is a service-level objective, or SLO. The important part is what gets counted, not the abbreviations.

What would count as getting the runbook?

For an invented internal search service, suppose the team wants 99% of eligible requests to return a successful result within 300 ms over a rolling 30 days. An empty result can be a valid answer; a server error cannot. These choices suit this example, not every search service.

Place the timer at entry to the search handler and stop it when the response is ready. It now includes dependency calls made inside that handler. It misses the journey to the handler and the delivery back to the caller. A client-side promise needs those intervals too. If the assistant spends seconds deciding to search, this timer cannot explain that earlier wait.

That ten-millisecond error is still bad: the responder never got a usable result. Settle what counts as completion and which requests belong in the population before writing the query. The SLO guide and Google’s implementation chapter explain why the measurement must be able to see the failures covered by the promise.

Follow a 120 ms search into the buckets

A 120 ms search increments the classic histogram’s 0.3-second bucket and every higher bucket. A 700 ms search does not enter the 0.3-second one. Each boundary counts all observations at or below it, so the buckets overlap rather than divide requests into separate piles. Add the buckets together and that quick search gets counted repeatedly. The histogram also exposes the total count and sum.

Give our example 10,000 completed requests: 9,850 fast successes, 100 slow successes and 50 failures. Only the first group meets both conditions. Dividing 9,850 by 10,000 gives 98.5%, leaving 150 bad events. A 99% target allows 100 bad events in this population, so these invented results use 150% of that allowance. Leaving the 50 errors out would make the missing answers disappear from the arithmetic.

A classic bucket exactly at 300 ms preserves the count we need; a percentile between boundaries requires an estimate. Prometheus’s histogram documentation now recommends native histograms where the collection path supports them. Existing classic bucket series still make this example useful. Understanding the promise need not wait for a format migration.

Make the first failed search visible

Before adapting the query, look at an actual scrape: the measurements Prometheus collects from the service. Find the metric name, its seconds unit and its outcome labels. The names below are invented for the example, not measurements a query can create. Instrumenting a real service requires permission and a compatible backend.

The Python wrapper below records one duration for each completed attempt. It uses two outcomes and explicit bucket boundaries. An exception means failure; a normal return means a valid response. If your application returns an error object normally, classify that as an error too, or the wrapper will congratulate it for returning.

from time import perf_counter
from prometheus_client import Histogram

SEARCH_SECONDS = Histogram(
    "runbook_search_duration_seconds",
    "Search handler duration, including unsuccessful attempts",
    ["outcome"],
    buckets=(0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.0),
)

# Expose both known outcomes before the first request.
SEARCH_SECONDS.labels(outcome="success")
SEARCH_SECONDS.labels(outcome="error")

def measured_search(query):
    started = perf_counter()
    outcome = "error"
    try:
        result = search_backend(query)
        outcome = "success"
        return result
    finally:
        SEARCH_SECONDS.labels(outcome=outcome).observe(
            perf_counter() - started
        )

Suppose the first search after startup is our ten-millisecond failure. The two earlier .labels() calls make the success series exist at zero before that request. Without them, no success series exists, which is different from observing zero successes. The Python client’s label guidance recommends this initialization. After enough scrapes for rates and an observed rise in error count within the window, the classic query can report 0% good requests. Initialization alone cannot manufacture traffic or replace missing scrapes.

This wrapper still needs the real search_backend and a metrics endpoint. It is synchronous scaffolding, not a working search service. The Python histogram reference describes the buckets and observations. For an asynchronous handler, surround the awaited lookup with the timer; timing the creation of a coroutine would stop before the search had finished.

An ordinary exception reaches finally, so this error gets recorded. Force-kill the process and that block cannot be relied upon to run. A request that never enters the handler is also invisible here. Compare the handler count with an independent ingress or client count during rollout. If those missing requests belong in the promise, measure somewhere that can count them; this ratio is not end-to-end reliability.

Count requests before comparing percentages

With the scrape label job="runbook-search" and the same bucket layout on every replica, the following query asks what fraction of observed requests succeeded within 300 ms in the last five minutes:

sum(rate(runbook_search_duration_seconds_bucket{
  job="runbook-search", outcome="success", le="0.3"
}[5m]))
/
sum(rate(runbook_search_duration_seconds_count{
  job="runbook-search"
}[5m]))

The numerator selects successes in le="0.3", the bucket at or below 0.3 seconds. The denominator takes both outcomes. First calculate each counter’s per-second rate, then add replicas, so resets can be handled for the individual series. Divide the two rates and the time unit cancels. The rate documentation explains why that order matters.

Five minutes tells us about recent behavior, not the rolling 30-day objective. For that longer view, replace both rate(...[5m]) expressions with increase(...[30d]), assuming the backend retains the history and can evaluate it. These are estimated counter increases. Keep the good and total events together across the window, then divide; averaging short-window percentages would give a quiet interval the same influence as a busy one.

Imagine two five-minute intervals. In the quiet one, all 10 of 10 searches are good. In the busy one, 9,800 of 10,000 are good. Averaging 100% and 98% produces an attractive 99%. Counting the actual requests gives 9,810 out of 10,010, about 98.002%, which misses the 99% target for those intervals. The busy period served a thousand times as many requests. It needs a thousand times the weight.

Keep request counts beside the percentage. No observations leave the ratio undefined, not 100%. A failed scrape raises another question: did requests stop, or did we stop seeing them? The missing-data guide follows that distinction.

You may also want to know how long the slow end of the successful searches took. This companion p99 query estimates the time at or below which 99% of successes fell:

histogram_quantile(0.99,
  sum by (le) (
    rate(runbook_search_duration_seconds_bucket{
      job="runbook-search", outcome="success"
    }[5m])
  )
)

It combines bucket observations before estimating the percentile rather than averaging replica percentiles. But it describes successes only. The SLI above keeps errors in its denominator. If slow requests increasingly fail, successful-search p99 can improve while the SLI gets worse. Both panels can be right; their labels need to tell the next responder which requests each includes.

The same promise in a native histogram

The same durations can travel as native histogram samples instead of separate floating-point bucket series. If the service and its whole telemetry path support that format, histogram_fraction can estimate the fraction of successful searches that were fast. This query assumes the base metric really contains native histogram samples. Renaming classic bucket series would not make it work:

histogram_fraction(0, 0.3,
  sum(rate(runbook_search_duration_seconds{
    job="runbook-search", outcome="success"
  }[5m]))
)
*
sum(histogram_count(rate(runbook_search_duration_seconds{
  job="runbook-search", outcome="success"
}[5m])))
/
sum(histogram_count(rate(runbook_search_duration_seconds{
  job="runbook-search"
}[5m])))

The responder still wants a successful search within 300 ms. The first term estimates the fast fraction among successes. Multiplying by the success rate and dividing by the all-outcome rate brings failed searches back into the total. A native bucket may straddle 300 ms, however, so the result can require interpolation, an estimate within that bucket. The function reference describes it. Its resolution and uncertainty must be suitable for the decision.

Now give that native query only our first failed search. With zero successful observations, histogram_fraction returns NaN, not a usable number; multiplying by a zero success rate does not produce the desired 0%. If the success histogram is absent, there may be no result instead. Classic label initialization does not solve this native case. Initialized counters for good and total events can provide the errors-only ratio while preserving the difference between no traffic and missing data.

For a fixed 300 ms promise, a good-event counter can directly record whether each search met it. That may be clearer when exact classification at a contractual boundary matters. Keep a histogram to investigate how durations changed. Neither job requires pretending the quick error was a good answer.

Try the measurement beside the existing alerts

Let the new calculation sit beside the current dashboard through a busy period before it influences paging. In a controlled test, restart the instrumented process and make the first search fail. With enough scrape samples, the initialized classic ratio should show 0% good requests, not lose its numerator. Then add a fast success and a slow one: only the fast success belongs in the good count, but all three attempts belong in the total. Follow with mixed traffic to check the same distinction beyond that tidy sequence.

A no-traffic window should stay undefined. Check scrape health separately: a range query may still contain older samples after a recent scrape failure. Agreement with the requests sent is the useful result here, even if the resulting percentage looks worse.

The following small experiment needs no running service. It checks the illustrative arithmetic and catches the tempting mistake of dividing only by successful requests:

fast_success = 9850
slow_success = 100
failed = 50
all_requests = fast_success + slow_success + failed
indicator = fast_success / all_requests
bad = all_requests - fast_success
assert indicator == 0.985
assert bad == 150
assert bad / (all_requests * 0.01) == 1.5
print(f"Good: {indicator:.1%}; bad events: {bad}")

This tests arithmetic, not PromQL execution or production coverage. During rollout, another distortion appears if only some replicas expose the scraped le="0.3" series: the numerator sees their fast successes while the denominator still includes everyone’s work. Check every replica. If the layout must change, use a versioned metric to compare the new population independently, and retire the old query only once the new one is complete.

Those extra measurements have a cost. Seven finite buckets, the infinite bucket, count and sum make ten core series per outcome per target, before additional client metrics. Two outcomes across twenty targets make 400. Extra labels multiply the number. Do not put raw search text, user IDs or request IDs in these labels; they expand the series population and may expose sensitive information. The instrumentation guidance discusses this growth. Keep detailed searches in appropriately controlled logs.

Watch added series, scrape payload size and query duration during the trial. A costly repeated query can become a recording rule, precomputed under an explicit name and evaluation interval. Retain the underlying counts for longer windows, not just the convenient percentage. If overhead is unacceptable or counts disagree with the independent request measurement, revert the instrumentation change while keeping existing alerts active.

The responder who got an error in ten milliseconds now appears in the total, but not among the good searches. That is where the missing runbook belongs. The 30-day fraction can inform the existing budget policy, with shorter-window alerts designed separately. Speed still matters; it just no longer gets to stand in for an answer.