Build the dashboard for the person waiting on a report

A customer refreshes an empty downloads page. On another screen, the CPU charts are green. Nothing about that pairing is impossible: the machines can have plenty of room to work while the report the customer wants remains unfinished.

Start the dashboard with that report’s journey. How much work arrived, how much finished, and what is still waiting? Machine measurements become useful companions when they help explain a change in those answers. They should not be the only answers available.

Use the lab’s /metrics output or the hypothetical counts below in a spreadsheet. A dashboard specification is a complete outcome for this exercise; a running Grafana instance is optional.

Ten arrived. Twenty finished. Nobody succeeded twice.

Arrange views for accepted exports, completions by outcome, pending work and suitable timing or deadline results. Label each with its region, interval and units. The local service supplies the first three and cumulative worker duration, but no latency histogram or deadline indicator. Its counters therefore cannot produce a p99 panel. To picture p99, line up measured durations from shortest to longest and find the duration about 99% of the way through that group, leaving roughly 1% above it.

Suppose ten exports arrive this minute and twenty finish. Dividing completions by arrivals gives 200%. That is not an unusually successful group of ten requests. Some completed work may have arrived earlier, while some of the new ten may still be running. The two counts compare throughput, meaning work moving in and out. They do not describe the success probability of one group.

To learn what happened to those ten new requests, follow the same ten until their outcomes are known. Meanwhile, the dashboard can show the flow: arrivals add demand, completions remove work, and pending count shows what remains. More completions alongside a shrinking queue suggest catch-up. A growing queue suggests work is entering faster than it leaves. Now CPU or storage measurements can help explain what is limiting progress.

Give region B room on the screen

Try a hypothetical ten-minute packet. Region A completed 9,900 exports without a failure. Region B completed 100, with 20 failures. The fleet failure fraction is 20 out of 10,000, or 0.2%. Region B’s is 20 out of 100, or 20%.

Both belong on the dashboard. The fleet total is accurate, but it offers little comfort to the smaller group whose work keeps failing. Selecting region B should carry that region and the same time window into logs or traces, not send the responder back to every service over the last day.

Make the picture match the question

A line can show a queue growing. A current number needs a unit. Latency needs a distribution or a fraction relative to a threshold. A lifetime counter displayed in a gauge will keep climbing during healthy operation; its movement alone can make normal accumulated work look like a worsening condition.

Try a panel title such as “Pending exports, current count” or “Completed exports per second, region B.” Put the query and metric in the description so another engineer can inspect the calculation. “Errors” leaves too much unresolved: failed API requests, worker attempts and completed exports are different populations.

Sketch the arrangement before choosing a visualization product. Keep related comparisons close and align their intervals. A responder should be able to look from arrivals to completions without mentally translating two different slices of time.

A drill-down should feel like moving closer to the same work. Keep its operation, region and interval, then add a job or trace identifier when available. If the next page drops those filters, the responder may spend several minutes explaining an event that never belonged to the original problem.

Can someone else find the delayed work?

Ask a partner to locate the affected region and explain whether arrivals, completions or both changed. Working alone, hide your notes and use the packet as a desk check. Add a case where telemetry stops: the display needs to distinguish no observations from observations showing zero failures.

Save each query, unit, population, interval and drill-down destination with the specification. A screenshot shows the arrangement; those details let the next engineer understand and reproduce what its panels mean.

Return to the empty downloads page. The sketch should now make room for work that arrived, finished, remains waiting or was not observed. The machine charts still have a job, but they no longer get to answer a question the customer never asked.

Sources

Prometheus instrumentation practices and Grafana dashboard practices provide implementation context. The sample counts are hypothetical.