The completed-job count dropped after a restart

After a process restart, a chart calculation appears to show a negative number of completed jobs. The synthetic Dispatch worker has not run backward. Its in-memory counter returned to its starting value, and the calculation treated that drop as negative work.

Here, a completed job means the lab worker waited and recorded a success or failure state. It does not mean a report was generated or delivered to a customer. The chart follows those simulated outcomes.

Follow the surprise through the query. In the synthetic Dispatch service, build views of completion throughput, failed completed jobs and pending work. The metrics are small enough to account for what each calculation includes and what it leaves out.

Use a Prometheus and Grafana lab you control, or work on paper. Dispatch exposes metrics at http://127.0.0.1:8090/metrics. If Prometheus is in a container, that container’s loopback address points to itself; use an approved route to the lab service instead.

Let each counter tell you about its own restart

For the fraction of completed simulated jobs that failed, use the supplied metric:

sum(rate(dispatch_exports_completed_total{outcome="failed"}[5m]))
/
sum(rate(dispatch_exports_completed_total[5m]))

Start inside the numerator. The selector keeps counters labeled outcome="failed". For each counter, rate(...[5m]) estimates the per-second change across five minutes, accounting for observed resets. Only then does sum add those rates together.

The denominator repeats that process for all completion outcomes, successful and failed. Dividing the two cancels the per-second units, leaving a fraction of completed work. With no completions or absent data, show the state explicitly; neither establishes that all exports succeeded.

Why take each rate first? Imagine one process restarting while another keeps completing jobs. Add their raw counters and the second process’s growth can conceal the first process’s drop. Calculating each rate before combining them preserves the reset information. The result is still a windowed estimate, not an exact timestamped list of completions.

An accepted job that disappears before completion never enters this metric. Name the panel “failed fraction of completed jobs” and keep accepted and pending work nearby. The panel should not quietly claim to account for jobs it never counted.

A regional legend cannot create regional data

In a larger service whose measurements include region, sum by (region)(...) on both sides can preserve that distinction. The local metric has no region label. Adding regional names to its legend would make the display look more informative without supplying the missing information.

Keep the operation, population and window visible, and pass them into linked logs or traces. If a larger service’s selected region disappears during navigation, a responder may start answering a different question without realizing the subject changed.

The same average can hide a very different wait

Dispatch records cumulative worker duration, not histogram buckets. Dividing its duration rate by the aligned completion rate can estimate a mean. But ten similarly quick exports can have the same total duration as nine quick ones and one very slow one. Their sum and count match; the waiting experience does not.

To picture p95 or p99, arrange measured durations from shortest to longest and find the duration about 95% or 99% of the way through the group. Roughly 5% or 1% lie above those respective positions. The lab’s totals cannot recover that ordering. A suitable histogram counts observations grouped by duration boundaries. Recording one is a prerequisite for this histogram-based calculation.

For a real classic histogram named export_duration_seconds_bucket, a conceptual fleet query is histogram_quantile(0.95, sum by (le)(rate(export_duration_seconds_bucket[5m]))). Here le retains the bucket’s upper boundary while counts are combined. The lab does not supply this metric. Boundaries must be compatible and observations sufficient; the resulting percentile is an estimate from those buckets, not a recovered list of exact durations.

Try the moments a smooth chart tends to hide

Predict or, in your controlled lab, observe four cases: restart the process, stop traffic, run with --fail-jobs and temporarily stop scraping. These produce reset, idle, failed-work and missing-observation situations. Replacing every awkward value with zero would make the pictures tidy by removing distinctions the responder needs.

A service that is scraped but completes no work supplies a zero-throughput observation. A target that is not scraped leaves a gap. Both can make a chart quiet, but only one tells you the service was observed doing no work.

Keep specifications for completion throughput, failed fraction of completed jobs and current pending count. Explain why a latency percentile is unavailable. Someone looking at an empty latency panel should not have to decide whether it means missing data or impossibly fast jobs.

Read each query as a sentence about the work and compare it with the title before polishing the display. The restart that first appeared to undo completed jobs should now have an understandable place in the chart. So should the silence when measurements stop arriving.

Sources

Prometheus functions, histograms, and Grafana Prometheus queries provide the query semantics.