Why a Prometheus error-rate query can return no data

A Prometheus error-rate query can return no data when its failure or total-request series is absent: the division has no matching value to return. An error-labeled counter may not have been initialized, collection may be lost, or labels may no longer match the selector. This is different from measured zero errors and from present but unchanged counters, whose zero-over-zero ratio is NaN. A display that fills these gaps with zero can make lost visibility look like recovery.

Before a team uses that ratio to allow a rollout or declare recovery, it needs to know whether requests succeeded, no eligible requests arrived, or the measurements are unavailable or incomplete. Each case calls for a different response. When measurements are missing, service health remains unknown until a responder checks another source; a quiet panel cannot resolve it.

A hypothetical checkout service makes the distinction visible in this guide. Its synthetic queries were tested with promtool 3.7.2; they were not exercised against a production service or notification pipeline. The point is to give the team fixed cases whose behavior can be inspected before attaching operational consequences to a chart.

Reproducing the checkout queries with promtool

The local exercise needs promtool 3.7.2, the downloadable synthetic fixture and the expressions being considered. It needs no production credentials or access. PromQL is the query language; promtool evaluates known sample sequences so a proposed change can be compared with an expected result rather than a reassuring dashboard color.

Choose one service and define an eligible request before interpreting the output. HTTP 5xx is only a transport-level definition of failure. An agent tool may return 200 for an incorrect change, so a generic HTTP counter cannot establish that an agent completed its task successfully. Keep those acceptance questions separate when applying the example.

What the ratio returns, and what an alert does with it

The example’s float counter, http_requests_total, accumulates request events. Labels identify the checkout job and response status; only 5xx responses count as failures here. The expression calculates the five-minute rate of those failures and divides it by the rate of all recorded requests:

sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))

The arithmetic is not yet an alert condition. Prometheus returns a vector of labeled values and activates an alert for each returned element, including one whose value is zero. Add a filtering comparison for the chosen threshold. A comparison using > bool retains false results as zero-valued elements, so it does not provide that filtering. Use the threshold comparison without bool, with timing and threshold selected for the service.

Calculate rate before aggregating so individual counter resets remain visible. The rate function reference explains why reversing that order loses information the rate calculation needs.

The result is limited to the population behind your metric. Requests that never reach the instrumentation, failures represented by a 200 response and unacceptable latency are outside this narrow indicator. They need appropriate measurement definitions rather than confidence borrowed from this ratio.

An empty vector creates no active alert. A for interval postpones firing; keep_firing_for can keep an already firing alert active after its condition disappears. These timing controls do not establish that lost measurements mean recovery. The alerting-rule semantics defines their behavior.

Remove one series and see what the arithmetic loses

The supplied local tests use one-minute samples and evaluate at minute ten. Their results below describe constructed input sequences, not observations from an incident:

Synthetic input	Error-ratio result	What the result establishes
Success counter increases; explicit error counter stays at zero	0	No errors in the supplied series
Success counter increases; error counter is absent	Empty vector	The expression cannot calculate the ratio as written
Both counters are absent	Empty vector	No ratio is available
Success counter increases by 54 and error counter by 6 each minute	0.1	Ten percent of the supplied request increments are errors

Notice the absent error counter in the second row. Some instrumentation creates an error-labeled series only after the first error; other instrumentation is expected to expose it beforehand. The same absence can therefore be normal initialization or a collection defect. Establish when each series should exist before deciding whether substituting zero has a defensible meaning.

Appending or vector(0) made the tested total-absence case return zero. That proves a change in query output, not successful requests. The set and matching operators explain how the fallback supplies an element when the ratio supplies none. Here that element has no labels, so it also cannot generally reconstruct missing service or region identity.

Present-but-unchanged counters reveal another case: zero divided by zero produces NaN, “not a number,” under PromQL floating-point rules. Independent review of the synthetic fixture also tested this. Quiet traffic may be expected, or may mean customers cannot reach checkout. Separate demand or reachability evidence is needed when those possibilities lead to different actions.

An absence test can cover the job and miss an instance

For the single named checkout job, this expression supplies a bounded starting measurement check:

absent_over_time(http_requests_total{job="checkout"}[5m])

The function reference defines the check as absence of matching samples over its range. It does not evaluate whether the sampled values describe success or failure.

In the local test, total absence returned {job="checkout"} 1. One surviving matching series instead produced no result. That is the limit of the selector: the job can still have observations while an instance, region or status series has disappeared.

An illustrative population check shows how misleading that surviving series can be. Over the same five-minute interval, suppose instance a records 100 requests and an explicit zero-valued error counter, while instance b records 100 requests with 20 errors. Complete measurement gives 20 ÷ 200, or 10%. If b disappears from the query window while a remains, the available measurements instead give 0 ÷ 100, or 0%. The job-wide absence check still returns nothing because a has samples. The zero now describes the observed subset; it cannot clear the missing instance or establish recovery for the whole job.

The fixture explicitly expects instance b, so a selector with instance="b" can identify its missing data. A changing fleet needs a changing expected population, including legitimate scale-down. Compare that inventory with observed coverage and give its maintenance an owner. A stale expectation can create misleading absence alerts just as a missing expectation can hide a lost instance.

The five-minute range is illustrative, not a universal response target. Even one matching sample within it prevents this absence expression from returning, so intermittent collection may need another coverage or freshness check. Instant selectors have distinct lookback and staleness behavior. Different expressions need not expose the same loss at the same moment.

What to do when checkout measurements disappear

Rehearse a service-owner policy that distinguishes what is known from what must be checked next:

Observation	Initial disposition	Evidence needed next
Valid population and elevated error ratio	Investigate customer failure	Affected operation, scope, recovery options
Valid population and zero errors	Continue normal checks	Traffic, latency, and excluded outcomes
No eligible traffic	Follow the service's quiet-traffic policy	Independent demand or reachability evidence
Missing or incomplete measurement	Record health as unknown	Collection state and independent customer evidence

For checkout, missing measurement should reach someone who can inspect collection and compare independent customer evidence. Whether that requires a page depends on the consequence of waiting and the independent coverage available. Naming the responder and the check during rehearsal turns “unknown” into owned work instead of leaving the on-call engineer to discover who can restore visibility.

That path must survive the failure it reports. A rule inside an unavailable monitoring system cannot notify anyone about that system’s own outage. Exercise an independent monitoring-and-delivery check too; a correct PromQL result establishes only one part of notification coverage.

If a release gate consumes the ratio, define its response to unknown health explicitly. Any allowed exception needs an owner, scope, expiry and verification step, so a missing sample does not quietly grant permission to proceed.

Keeping the query cases as regression tests

Download the six-case test fixture and run promtool test rules prometheus-missing-data-test.yml with promtool 3.7.2 to reproduce the local checks.

Store synthetic inputs and expected outputs with the alert using promtool's rule-test format. Extend these cases with relevant resets, stale markers, label changes, quiet traffic, recovery and the timing clauses actually configured. That preserves the behavior the team intends through later edits.

The initial five cases covered nine expression results. Independent review reproduced those and added a sixth case with three checks for zero-traffic NaN and filtering versus boolean comparisons. They do not cover real scrape failures, firing delays, Alertmanager routing, dashboard rendering or production cardinality. Those require separate evaluation rather than being inferred from a passing fixture.

Keep the reasoning with the tests: when the counter exists, which instances are expected, how much absence can be tolerated and who can inspect independent evidence. Name the case that would make the ratio unsafe for recovery. Beside the incident dashboard, that short explanation helps a responder interpret the output instead of guessing why a zero or gap was considered acceptable.

From query tests to alerts in staging

After local tests pass, evaluate the proposed rules in authorized staging. Check actual labels and query duration before attaching notifications or a release decision. Rehearse lost and restored collection separately from application failure and verify the recipient of each signal. The different states should remain distinguishable outside the query engine.

Version the old rule and dashboard so a misleading or expensive pilot can be reversed. Preserve the agreed independent measurement-loss check during rollback. An unresolved limitation should remain explicit; converting unknown to healthy to reduce noise would undo the distinction the pilot was meant to establish.

Measure query duration, series count, alert volume and the time required to resolve the synthetic cases. Broad selectors and high-cardinality labels add work, so scope the trial and avoid request-ID labels. Keep secrets and customer data out of series and annotations. If the measurement contract or owner of unknown health remains undefined, settle that before relying on the ratio for rollout. A zero becomes useful evidence only when the team knows what was actually measured.