The alert fired. Did anyone receive it?

At 14:04, the condition turns red. The intended responder’s inbox stays quiet. In this hypothetical test, the calculation can be perfectly correct and still stop short of the person who needs to act.

Follow the alert all the way there. A paper exercise can establish when it should fire; an optional controlled local route can test whether the message arrives and helps someone find the affected work. Those are different accomplishments, and both matter.

Use an alerting system you control or stay with the supplied paper packets. The local service supplies teaching counters, not a production service-level objective, or SLO. Real paging requires an agreed exercise window and recipients.

Start with the exports that failed

Our hypothetical policy asks for immediate attention when confirmed export failures threaten the agreed objective quickly enough. Its condition divides failed completed exports by all completed exports over a named interval. Before using that fraction, decide what happens when there are very few completions or no data at all.

For the optional local route, restart Dispatch with --fail-jobs, send a small number of POSTs to /exports and inspect /metrics. Failure counts should rise after the configured worker delay. A restart clears counters, so account for resets rather than subtracting snapshots taken from different process runs.

A red condition, a quiet condition and an unanswered question

Use a training rule of more than 10% failures with at least 50 completions. With 100 completions and no failures, the fraction is 0%: no alert. With 100 completions and 20 failures, it is 20% and the population is large enough: the rule should fire.

The third packet contains no samples. There is nothing observed to divide. Calling that 0% would turn missing observation into a reassuring result; the answer is unknown.

One failure in one completion gives a fourth useful comparison. It is 100%, but this particular rule would not page because the population is below 50. That is an exercise policy, not proof that small populations do not matter. A low-volume service doing high-consequence work may need a different signal.

Now read the proposed message. It should name the operation, population and interval, with links to the owner, first useful check and escalation route. Open the investigation link: does it retain the same region and window? The responder should not have to rediscover which work made the rule fire.

Meet the message at its destination

On an authorized local route, use a test receiver or agreed non-production channel. Trigger the controlled failure and record four times: when the job failed, when the query met its condition, when the message was emitted and when it arrived. Then acknowledge it. Check whether grouping tucked it into an older notification or a silence suppressed it.

Those times tell you where waiting occurred. The application can fail before the next rule evaluation, and notification grouping can introduce another delay. A single arrival timestamp cannot distinguish the two.

Read what actually arrived as someone who did not build the rule. Can you find the affected work, matching records and owner? “High errors” can be delivered flawlessly while leaving almost the entire investigation to the recipient.

Restore normal lab mode, send successful work and look for a recovery message if the system supports one. A cleared condition is not a receipt for every earlier export. Check those older jobs’ outcomes separately.

Now let the observation disappear

In the lab, stop the service or stop scraping, the collection of its measurements. Does the system report an absent series, keep an old value or evaluate a separate availability rule? Decide who should know that this observation is missing. Filling the gap with zeros would invent successful work.

Show where the journey stopped

Keep the query, sample packets, route configuration, received messages and timestamps together. Paper arithmetic establishes a condition decision. A received message establishes delivery. One cannot stand in for the other.

If the rule fired and the receiver stayed quiet, follow the route. If the message arrived but its link opened the wrong region, repair that handoff into the investigation. The red condition at 14:04 is a beginning. The test becomes useful when you can say how far its warning traveled and whether the person at the other end could use it.

Sources

Prometheus alerting rules and Alertmanager document evaluation and delivery. The threshold and sample packets are illustrative.