On this page5 sections
A responder opens Grafana after hearing that customers cannot finish checkout in one region. The chart across the top looks reassuring: response times across the fleet are normal. The complaint has not gone away.
In this hypothetical incident, the chart and the customer can both be telling the truth. The chart covers more requests than the ones now failing. The dashboard needs to help the responder narrow that view, then keep the same customers and time interval in sight as the investigation moves elsewhere.
That is the useful design problem. Someone joining halfway through should be able to find the affected operation, see how much traffic it represents and follow a plausible lead to the resources serving it. Adding another attractive panel cannot correct a query that includes the wrong requests.
Find the checkout requests inside the fleet view
Start with an authorized Grafana installation and a data source such as Prometheus. Grafana displays the result of a query; the source supplies the measurements. In our example, the broad chart may be displaying its answer perfectly well. It is the question behind that chart that needs narrowing.
Check which requests the query includes before changing the chart's colors. Give the responder a clear route to the affected region, and make sure the on-call rotation can open the links that lead onward. A permission error at the log link would end the investigation just as effectively as a missing panel.
Grafana's dashboard guidance recommends organizing around a question. A service responder may begin with failed checkouts while an infrastructure responder begins with a machine. Their views should connect. Keep the fleet summary small, with room to explore a question that its designer did not anticipate.
Give the first panel the customer's problem
Create a dashboard and add a panel. Select the data source, enter its query and refresh the result. Choose a suitable visualization, name the measurement in its title and description, and save the dashboard. Grafana's creation guide covers the current interface and required permissions.
For checkout, the first view needs failures or response times together with request volume. Add the way to narrow by region before filling the surrounding space with resource charts. If nobody can yet explain the application metrics, find their owners and meaning first. A CPU chart tells you about processor activity; it cannot supply the missing checkout measurement.
Choose a known incident and fix the start and end times, rather than using a moving range such as the latest few minutes. Does the panel reveal the symptom customers experienced? Let the next panel answer something about those same requests during that same interval. Empty space on the dashboard is not a question that needs answering.
The busiest machine may be somewhere else
Suppose the regional view reveals failures that were hidden in the fleet summary. Before chasing the highest CPU value anywhere, identify the instances serving those requests. The following Prometheus query gives a non-idle CPU percentage for each instance. It does not select a region for you:
100 * (1 - avg by (instance) (
rate(node_cpu_seconds_total{mode="idle"}[5m])
))
The counter records accumulated idle CPU time. Taking its rate over five minutes estimates how much idle time accumulated per second. Averaging by instance combines that instance's CPU measurements; subtracting the idle share from one and multiplying by 100 gives the non-idle percentage.
That name matters. Non-idle includes I/O wait, time accounted for waiting on input or output, rather than only time executing application instructions. Keeping the instance label also lets a busy host remain visible instead of blending into the fleet average.
Now imagine that the busiest instance serves an unaffected region. The instances serving failed checkouts show no matching CPU rise. Following the brightest CPU panel would send our responder away from the complaint. Compare failures, response times and how close resources are to their limits for the relevant instances first. A busy host may be doing expected work. This is an illustrative comparison, not a measured result or a reason to dismiss every possible CPU problem.
Perhaps the affected instances have less memory available during the failing interval. This second query gives the responder a way to examine that possibility:
100 * node_memory_MemAvailable_bytes
/ node_memory_MemTotal_bytes
Available bytes divided by total bytes, multiplied by 100, gives the percentage of memory available. These are current quantities, so this calculation does not need a rate. A fall beside the checkout failures would give the responder a lead, not establish that memory caused them. Reading the result as bytes per second would describe something else entirely.
The calm fleet response-time chart deserves the same care. A histogram counts observations in buckets, such as ranges of request duration. Check those buckets and which requests have been grouped together before calculating a percentile, a value describing a chosen position in the ordered durations. Averaging percentiles already calculated for separate regions or instances does not produce the fleet's percentile. The regional complaint needs a calculation over the relevant requests, not an average of convenient summaries.
Follow the requests from symptom to investigation
The conceptual route starts with the user symptom, narrows the traffic being examined, compares dependencies and resources during the same period, and reaches individual request records. It depicts a reading order, not a measured incident. Diagram labels: User symptom: Errors or latency for the affected operation; Traffic + population: Region, tenant or other relevant scope; Dependency and resource signals: Compare against the same period; Logs or traces: Inspect the requests behind the aggregate.
When the error line disappears
The regional error line disappears. It is tempting to call that recovery, but there are other explanations: requests may have stopped arriving, or collection may have stopped. Traffic and measurement freshness help separate those possibilities. The blank panel alone cannot.
Include a missing target and a failed query in the dashboard trial, alongside a service that is genuinely quiet. The responder should be able to recognize the difference instead of having to remember which kind of blank is reassuring.
A deployment marker near the first failure is another useful lead. Check the version and region before following it: did that change reach the requests now failing? Two events being close in time helps you choose where to look; it does not establish a cause.
Then follow the checkout log link. Where supported, it should carry the service, environment and incident interval with it. If yesterday's regional failure opens as the latest fifteen minutes, healthy current requests may seem to contradict the old chart. The link has opened successfully while quietly moving the investigation to another time.
Hand the dashboard to the next person
Give another responder a known incident and its fixed start and end times. Ask them to reach the affected requests from the fleet summary, compare values and units with direct source queries, and open the matching records. Include missing data and the rotation's actual restricted permissions. A log link they cannot open breaks the same route as one that drops the time window.
Keep that route intact when the dashboard changes. Version the queries and review changes in what they count. Otherwise removing requests from a query can look like checkout improved. Choose the refresh interval with similar care: asking the source more often can add load without providing new information. The metrics guide explains counters, gauges and distributions in more detail.
Our hypothetical responder still has a cause to investigate. But now the investigation follows the failing checkouts: the wrong region's busy host has stayed out of the way, the missing line has raised a question instead of ending one, and the log link has kept the right interval. The dashboard has helped the next person pick up the incident where the customers are experiencing it.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- dashboard guidancegrafana.com
