When the checkout error line disappears

The checkout error line disappears just as someone asks whether the rollout can continue. “Looks clean,” comes the reply. Perhaps the failures stopped. Perhaps requests stopped. Perhaps the measurements stopped. The panel has become quieter, but the responder still has three very different situations to sort out.

A hypothetical checkout service makes those differences visible. A measured zero describes the requests that were observed. With no eligible traffic, there is nothing to divide. With missing or incomplete measurements, service health remains unknown until another source helps establish it. Before that ratio can announce recovery or permit a release, the unknown state needs an owner and an independent check.

The synthetic queries below were previously tested with promtool 3.7.2. Those bounded results are preserved here; they were not production-service or notification-pipeline tests. Known inputs let you examine what a query means before attaching a real decision to it.

Give the query a checkout you can inspect

The local exercise uses promtool 3.7.2, the downloadable fixture and the proposed expressions, without production access or credentials. PromQL is Prometheus's query language. Promtool can feed it known sample sequences and compare its answers with expected results, so a disappearing line becomes something you can reproduce.

First decide which requests count. Here, failures are HTTP 5xx responses. A wrong answer returned as HTTP 200, or a request that takes too long, needs a separate indicator. The denominator is the request population being counted; it cannot speak for outcomes outside the population behind the metric.

A zero-valued result can still activate an alert

The float counter http_requests_total accumulates requests, with job and response-status labels. The expression below divides checkout's five-minute 5xx rate by the five-minute rate of all its recorded requests:

sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))

Calculate the rate of each counter before adding them together. Otherwise a reset in one counter can be hidden by the aggregate, as the rate reference explains. If this expression returns zero, it has found no errors in the supplied series. It has not shown that every customer operation succeeded.

Turning the ratio into an alert introduces another distinction. Prometheus considers an alert active for each returned vector element, including a zero-valued one. A vector is the set of labeled results returned by the expression. Use a filtering comparison, such as a chosen > threshold without bool, to remove results that do not meet the condition. With bool, a false comparison can remain as a zero-valued element and still activate the alert. The service needs its own threshold and timing.

The for setting delays firing while the condition remains active. keep_firing_for can keep an already firing alert active after that condition disappears. The alerting-rule reference describes both. Neither setting turns lost measurements into proof of recovery, and an empty vector supplies no active alert element.

Take away the error counter

The fixture supplies samples one minute apart and evaluates the expression at minute ten. These constructed inputs show how a few apparently similar cases produce different answers:

Input supplied	Ratio returned	What it tells you
Success counter rises; an explicit error counter remains zero	0	No errors appear in those supplied series
Success counter rises; the error counter is missing	Empty vector	This expression cannot produce the ratio
Neither counter is present	Empty vector	There is no available ratio
Each minute adds 54 successes and 6 errors	0.1	Errors account for ten percent of supplied request increments

The missing error counter deserves a closer look. Some instrumentation creates it only when the first error occurs. Other instrumentation promises an explicit zero from startup. The same blank result can therefore reflect normal initialization or lost data. What the instrument promised to publish matters before you choose a fallback.

In the test, adding or vector(0) made total absence return zero. No successful request appeared in the meantime; only the result changed shape. Under the set and matching rules, this supplies an unlabeled element where the ratio supplied none. It cannot generally rebuild the identities of missing services or regions either.

Keep both counters present but unchanged and the arithmetic changes again: zero divided by zero produces NaN, meaning not a number. The fixture's previous independent review tested this case too. A shop with no customers may legitimately have no requests; customers unable to reach checkout can leave the same traffic gap. Independent demand or reachability information separates those situations when the response depends on it.

One reporting instance can make the job look present

For the single named checkout job, this small measurement check asks whether any matching samples arrived within the selected range:

absent_over_time(http_requests_total{job="checkout"}[5m])

absent_over_time returns a result when matching samples are absent throughout that range. It does not judge whether requests succeeded. In the fixture, complete absence returned {job="checkout"} 1. One surviving matching series was enough to leave the result empty.

Imagine instance a reports 100 requests with no errors, while b reports 100 requests with 20 errors over the same interval. Together that is 20 out of 200, or 10%. Lose b's measurements and the available ratio becomes zero out of 100. The job-wide absence check stays quiet because a is still reporting. Checkout appears healthier precisely when part of it has disappeared from view.

The fixture knows b should exist, so a selector containing instance="b" can detect its absence. In a changing fleet, that expectation needs a maintained inventory and an owner, including legitimate scale-down. Expect a retired instance forever and you create a false alarm; forget an active one and you can hide its loss.

The five-minute range is illustrative. A single matching sample inside it prevents this absence expression from returning, so intermittent collection may need a separate freshness or coverage check. Instant selectors have their own lookback and staleness behavior. Different expressions need not reveal the same gap at the same time.

Make unknown lead to a person and a useful check

What is visible	Where to begin	What to establish next
Valid population with an elevated error ratio	Investigate customer failure	Affected operation, extent and recovery options
Valid population with zero errors	Continue normal checks	Traffic, latency and outcomes this metric excludes
No eligible traffic	Use the service's quiet-traffic policy	Independent demand or reachability information
Missing or incomplete measurements	Keep health marked unknown	Collection state and an independent view of the customer path

For lost checkout measurements, identify someone who can inspect collection and compare an independent customer-path check. Whether they need a page depends on the consequence of waiting and the coverage that remains. Decide that during rehearsal, rather than asking an incident responder to discover who owns a blank panel.

The two checks answer different questions. Inspecting collection can explain where the records went. Trying the customer path can show whether checkout works now. A healthy collector does not settle the customer question, and one successful checkout does not explain the missing records. Together they give the responder somewhere to go.

The notification path needs independent protection too. A rule inside an unavailable monitoring system cannot announce that system's outage. Exercise both independent monitoring and delivery, not only the expression. If a release gate consumes the ratio, define what it does when health is unknown. Any exception needs an owner, scope, expiry and verification, rather than letting missing samples silently authorize progress.

Keep the strange cases where the next editor will find them

Download the six-case fixture and run promtool test rules prometheus-missing-data-test.yml using promtool 3.7.2. Keep the synthetic inputs and expected outputs alongside the rule in promtool's test format, so a later edit can be checked against the same cases.

The original five cases covered nine expression results. Previous independent review reproduced them and added a sixth case with three checks for zero-traffic NaN and filtering versus boolean comparisons. That history does not demonstrate real scrape-failure behavior, firing delays, Alertmanager routing, dashboard rendering or production cardinality, the number of distinct time series. Extend the cases for relevant resets, stale markers, label changes, quiet traffic, recovery and the timing clauses actually used.

Beside the incident dashboard, explain when the counter exists, which instances should report, how long missing data can be tolerated and who performs the independent check. The next person changing the query should be able to see why a pleasing zero is not always the answer the service needs.

Rehearse the response before it controls a rollout

In staging, use the real label shapes and follow both routes. A collection failure should reach the person responsible for lost visibility; an application failure should follow its response path. Restore collection and check that a reported zero is supported by current samples again. Include query duration and delivered notifications before either result is allowed to govern a rollout.

Keep versions of the previous rule and dashboard so a misleading or costly pilot can be reversed. Retain the independent measurement-loss check during rollback. Silencing noise by making unknown look healthy would remove the distinction the exercise was meant to preserve.

When that checkout line next disappears, the responder should know which question comes next: did traffic stop, did coverage fail, or did the measured failures end? Carry those meanings from the query through the dashboard to the release decision. The screen can stay simple without asking the person watching it to guess.