Build a Grafana dashboard that follows the incident

In a hypothetical incident, the fleet latency panel looks normal. Checkout is failing in one region. Both can be true, but the responder needs a path from the broad summary to those customers before the dashboard becomes useful.

Design that path for someone arriving halfway through the incident. They need the affected operation, enough traffic context to judge the result, and a way to find the resources or dependencies that could explain it. Attractive panels cannot fix a query that counts the wrong requests.

Start with the operation the user cannot finish

The responder needs an authorized Grafana installation and a data source such as Prometheus. In our checkout example, Grafana displays the result of a query; the source supplies the measurements. A normal-looking fleet panel might be faithfully displaying a population too broad to answer the regional complaint. Before changing its colors, look at which requests the query includes. The rotation also needs permission to follow the evidence links.

Grafana’s dashboard guidance organizes around a question. A service responder and an infrastructure responder may start in different places, but their views should connect customer symptoms to the resource investigation.

The fleet summary can remain small. What matters here is an obvious route to the failing region, with an exploratory workspace available when the responder needs a question the summary did not anticipate.

Give the first panel a job

Create a dashboard, add a panel, select the data source and enter its query. Refresh, choose a suitable visualization, supply a title and description naming the measurement, and save. Grafana’s creation guide covers current interface and permission details.

For this incident, start with checkout failures or latency and request volume. Add a way to narrow region before surrounding the view with resource charts. If the application metrics are unknown, find their owners and meaning first; a CPU graph cannot substitute for them.

Use an absolute interval around a known incident. Does the panel show the symptom people experienced? Give the next panel a question about those same requests in that same interval, rather than asking what would fill the remaining space.

Read the units before the color

Suppose narrowing to the affected region reveals checkout failures that the fleet view hid. The next comparison is not “which host is busiest anywhere?” but “what changed on the instances serving these requests during this interval?” After identifying those instances, this Prometheus resource query supplies a non-idle CPU value for each instance. The query itself does not select a region:

100 * (1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
))

The counter accumulates idle CPU time. Its five-minute rate and per-instance average estimate the idle share; subtracting it from one and multiplying by 100 gives the non-idle percentage. Non-idle includes I/O wait, so it is not simply time executing application instructions. Keeping instance labels prevents a busy host disappearing into a fleet average.

Imagine the busiest instance belongs to an unaffected region, while the affected instances have no corresponding CPU rise. Paging on the brightest CPU panel would take the responder away from the complaint. Compare failures, latency and saturation for the relevant instances first; a busy host may simply be doing expected work. This is an illustrative comparison, not a measured result or a reason to rule out every CPU problem.

The affected instances might instead have less memory available during the failing interval. This query lets the responder examine that possibility rather than infer it from CPU color:

100 * node_memory_MemAvailable_bytes
    / node_memory_MemTotal_bytes

Available and total bytes are current quantities, so their ratio gives available memory share without a rate. A fall alongside failures would be a lead to investigate, not proof that memory caused them. Reading it as bytes per second would answer a different question entirely: units need to agree before values are combined.

Returning to the deceptively calm fleet latency panel, inspect its histogram buckets and grouping before calculating a percentile. Averaging existing instance or region percentiles does not produce a fleet percentile. The responder needs a correctly calculated view of the affected requests, not a reassuring number assembled from incompatible summaries.

An incident dashboard needs a route through the evidence

Read diagram description

Suggested reading order. Start with the affected user operation, then narrow by traffic and population before investigating resources. Check freshness throughout. Diagram labels: User symptom: Errors or latency for the affected operation; Traffic + population: Region, tenant or other relevant scope; Dependency and resource signals: Compare against the same period; Logs or traces: Inspect the requests behind the aggregate.

Show when the view cannot answer

Suppose the affected region’s error line then disappears. Have checkouts recovered, have requests stopped arriving, or has collection stopped? Traffic and freshness distinguish those possibilities; the empty panel alone cannot. A dashboard trial should include a disappeared target and failed query as well as a genuinely quiet service, so this moment does not become false reassurance.

A deployment marker near the first failure makes a useful lead. The responder still needs version and region context: did that change reach the requests now failing? Temporal proximity helps locate a possible cause; it does not prove one.

Following the checkout log link should retain service, environment and incident interval where supported. If yesterday’s regional failure opens as the latest fifteen minutes, perfectly healthy current requests can appear to contradict the old panel. The link works, but the investigation has lost its time window.

Walk the dashboard as the next responder

A useful rehearsal is to give the next responder a known incident and its absolute time range. Can they reach the affected population from the fleet summary, compare panel values and units with direct source queries, then open the matching records? Missing data and the rotation’s actual restricted permissions belong in that rehearsal too. An inaccessible log link would break the same journey as a lost time window.

The regional comparison also needs to survive dashboard edits. Version queries and review changes in meaning, so a changed population is not mistaken for improved checkout. Refreshing more frequently can add source load without supplying new information about those requests; choose the interval accordingly. The metrics guide explains counters, gauges and distributions in more detail.

Our hypothetical responder has not proved a root cause. They have avoided a busy host from the wrong region, recognized that a missing line is not recovery, and kept the failing requests’ time window on the way to logs. That is real progress for a dashboard: the next question concerns the customers who cannot check out, not whichever panel looks most dramatic.