Prometheus missing data: test what your error-rate alert cannot see

An error-rate alert can stop producing results when the telemetry needed to calculate it disappears. If the dashboard replaces that gap with zero, the incident view can look reassuring precisely when its evidence has weakened.

Before using an error rate to declare recovery or allow a rollout, test three states separately: measured success, no eligible traffic, and unavailable measurement. They need different interpretations and may need different owners.

This guide uses a small, hypothetical checkout service to expose those differences. The queries were tested with synthetic series in promtool 3.7.2. They have not been exercised against a production service or notification pipeline.

Keep the first pilot local

Use promtool 3.7.2, the downloadable synthetic fixture and a copy of your proposed expressions. No credentials or production access are needed for these local checks. PromQL is Prometheus’s query language; promtool can evaluate fixed sample sequences so a query change has an inspectable expected result.

Start with one service and its agreed definition of an eligible request. A generic 5xx counter cannot establish the success of an AI agent’s work: a tool can return HTTP 200 while the requested change is incorrect. For an agent-facing service, keep task acceptance and this transport-level indicator separate.

Start with the expression behind the color

Assume a float counter called http_requests_total, a job label of checkout, and HTTP status labels. For this example only, HTTP 5xx responses count as failures:

sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))

This ratio is an indicator, not a complete alert rule. A rule needs a filtering condition, such as a service-specific threshold comparison without the bool modifier. A raw ratio of zero still returns a vector element and therefore activates an alert; > bool also retains false results as zero-valued elements. Define the threshold and timing from your service policy.

Apply the rate before aggregation so individual counter resets remain visible to the calculation. Prometheus documents that behavior in its rate function reference.

This is a deliberately narrow indicator. Requests that never reach this instrumentation, application failures returned with a 200 status, and unacceptable latency need their own measurement definition. Confirm the population behind your metric before adapting the query.

A Prometheus alert expression becomes active when it returns vector elements; an empty result does not create an active alert. A configured for period delays firing, while keep_firing_for can extend an already firing alert after its condition disappears. Neither establishes that missing measurements represent recovery. See the alerting-rule semantics.

Reproduce the gap before choosing a fallback

The local tests supplied one-minute samples and evaluated the expression at minute ten. These are constructed inputs, not incident observations.

Synthetic input	Error-ratio result	What the result establishes
Success counter increases; explicit error counter stays at zero	0	No errors in the supplied series
Success counter increases; error counter is absent	Empty vector	The expression cannot calculate the ratio as written
Both counters are absent	Empty vector	No ratio is available
Success counter increases by 54 and error counter by 6 each minute	0.1	Ten percent of the supplied request increments are errors

The absent-error case deserves investigation. Some instrumentation does not expose a labeled series until that outcome occurs. Missing errors might mean none occurred, or that instrumentation is broken. Establish the metric's contract before filling the gap.

Appending or vector(0) to the complete ratio made the total-absence test return zero. That is a tested change in query output, not evidence of success. PromQL's set and matching operators explain why the fallback can supply an element when the ratio supplies none. It also supplies an unlabeled element here; it is not a general way to restore missing service or region labels.

Keep zero traffic separate. With both counters present and unchanged, the arithmetic is zero divided by zero, yielding NaN under PromQL's floating-point rules. Independent review also tested this zero-traffic case. Decide whether quiet traffic is expected, and whether an independent request source is needed to distinguish quiet demand from an unreachable service.

Add a bounded measurement check

For this single, explicitly named job, a starting check is:

absent_over_time(http_requests_total{job="checkout"}[5m])

The function reference defines this as detecting the absence of matching samples over the range. It does not inspect whether their values indicate success.

The local test returned {job="checkout"} 1 when all matching series were absent. It returned nothing when one matching series survived. That second result is the important boundary: losing one instance, region, or status series can escape a job-wide check.

For stable, explicitly expected instances, a selector including instance="b" detected the missing instance in the fixture. Do not hard-code that pattern across an ephemeral fleet and assume it will stay correct. Compare the intended target population with observed coverage, and define how legitimate scale-down changes the expectation. The source of that expectation also needs an owner and a freshness check.

Five minutes is an illustrative tolerance, not a recommended response target. A single matching sample within the range prevents this absence expression from returning a result. Intermittent collection can therefore require a separate coverage or freshness check. Instant selectors have their own lookback and staleness behavior; do not assume every gap becomes visible at the same instant.

Give the unknown state a response

Treat the following as a proposed operational policy to rehearse with your service owner:

Observation	Initial disposition	Evidence needed next
Valid population and elevated error ratio	Investigate customer failure	Affected operation, scope, recovery options
Valid population and zero errors	Continue normal checks	Traffic, latency, and excluded outcomes
No eligible traffic	Follow the service's quiet-traffic policy	Independent demand or reachability evidence
Missing or incomplete measurement	Record health as unknown	Collection state and independent customer evidence

Route measurement loss to someone who can investigate it. Whether that warrants a page depends on the consequence of waiting and the available independent coverage. Avoid generating an urgent interruption with no reachable responder.

A rule running inside an unavailable monitoring system cannot notify you about that system's own outage. Rehearse an independent check of the monitoring and delivery path as well. Query correctness alone cannot establish end-to-end notification coverage.

If a release gate consumes this indicator, specify how it handles unknown health. Record any permitted exception with its owner, scope, expiry, and verification step.

Leave a repeatable check for the next change

Download the six-case test fixture and run promtool test rules test.yml with promtool 3.7.2.

Use promtool's rule-test format to preserve synthetic series and expected outputs alongside the alert. Start with the cases above, then add counter resets, stale markers, label changes, zero traffic, recovery, and any alert timing clauses you actually use.

The original five cases checked nine expression results. Independent review reproduced them and added a sixth case with three checks covering zero-traffic NaN and filtering versus boolean comparisons. They did not test scrape failures, rule firing delays, Alertmanager routing, dashboard rendering, or production cardinality. Those are separate checks.

Finish the review with a short record: metric contract, expected population, missing-data tolerance, independent evidence source, response owner, and the test that would invalidate the current design. Put a link to that record beside the incident dashboard, where the next responder can find it.

Introduce the rule with a way back

After the local fixture passes, evaluate the proposed expressions in an approved staging environment. Inspect the actual label sets and query duration before connecting the rule to notifications or a release gate. Rehearse missing and restored collection separately from a real application failure, and verify who receives each signal.

Keep the prior rule and dashboard configuration versioned. If the pilot produces misleading states or excessive evaluation load, restore that configuration and retain the independent measurement-loss check agreed with the service owner. Record the unresolved limitation instead of changing unknown to healthy to silence the noise.

Measure evaluation duration, series count, alert volume and time needed to resolve the synthetic cases. Wide selectors and high-cardinality labels increase processing and storage work; scope the pilot rather than adding request identifiers as labels. Keep tokens, customer data and credentials out of test series and alert annotations. If the service lacks a stable measurement contract or an owner for unknown health, finish that work before using this ratio to authorize a rollout.