Opens in a new tab
Guide

Follow the failing checkouts through your Grafana dashboard

In brief

A calm fleet chart can hide a regional checkout failure. Keep the affected requests and their time window together as you move from the first panel to resource queries and logs.

7 min read

Is this the right starting point?
Before you read
Basic monitoring and Prometheus-query familiarity, plus authorized access to Grafana and a configured data source.
What you will take away
Build and validate a symptom-led dashboard that preserves population, units, freshness and investigation context.
Sources
A large translucent blue lens spans four cream paper strips, while a smaller brass magnifying glass sits over a coral-colored patch on one strip.
On this page5 sections

A responder opens Grafana after hearing that customers cannot finish checkout in one region. The chart across the top looks reassuring: response times across the fleet are normal. The complaint has not gone away.

In this hypothetical incident, the chart and the customer can both be telling the truth. The chart covers more requests than the ones now failing. The dashboard needs to help the responder narrow that view, then keep the same customers and time interval in sight as the investigation moves elsewhere.

That is the useful design problem. Someone joining halfway through should be able to find the affected operation, see how much traffic it represents and follow a plausible lead to the resources serving it. Adding another attractive panel cannot correct a query that includes the wrong requests.

Find the checkout requests inside the fleet view

Start with an authorized Grafana installation and a data source such as Prometheus. Grafana displays the result of a query; the source supplies the measurements. In our example, the broad chart may be displaying its answer perfectly well. It is the question behind that chart that needs narrowing.

Check which requests the query includes before changing the chart's colors. Give the responder a clear route to the affected region, and make sure the on-call rotation can open the links that lead onward. A permission error at the log link would end the investigation just as effectively as a missing panel.

Grafana's dashboard guidance recommends organizing around a question. A service responder may begin with failed checkouts while an infrastructure responder begins with a machine. Their views should connect. Keep the fleet summary small, with room to explore a question that its designer did not anticipate.

Give the first panel the customer's problem

Create a dashboard and add a panel. Select the data source, enter its query and refresh the result. Choose a suitable visualization, name the measurement in its title and description, and save the dashboard. Grafana's creation guide covers the current interface and required permissions.

For checkout, the first view needs failures or response times together with request volume. Add the way to narrow by region before filling the surrounding space with resource charts. If nobody can yet explain the application metrics, find their owners and meaning first. A CPU chart tells you about processor activity; it cannot supply the missing checkout measurement.

Choose a known incident and fix the start and end times, rather than using a moving range such as the latest few minutes. Does the panel reveal the symptom customers experienced? Let the next panel answer something about those same requests during that same interval. Empty space on the dashboard is not a question that needs answering.

The busiest machine may be somewhere else

Suppose the regional view reveals failures that were hidden in the fleet summary. Before chasing the highest CPU value anywhere, identify the instances serving those requests. The following Prometheus query gives a non-idle CPU percentage for each instance. It does not select a region for you:

100 * (1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
))

The counter records accumulated idle CPU time. Taking its rate over five minutes estimates how much idle time accumulated per second. Averaging by instance combines that instance's CPU measurements; subtracting the idle share from one and multiplying by 100 gives the non-idle percentage.

That name matters. Non-idle includes I/O wait, time accounted for waiting on input or output, rather than only time executing application instructions. Keeping the instance label also lets a busy host remain visible instead of blending into the fleet average.

Now imagine that the busiest instance serves an unaffected region. The instances serving failed checkouts show no matching CPU rise. Following the brightest CPU panel would send our responder away from the complaint. Compare failures, response times and how close resources are to their limits for the relevant instances first. A busy host may be doing expected work. This is an illustrative comparison, not a measured result or a reason to dismiss every possible CPU problem.

Perhaps the affected instances have less memory available during the failing interval. This second query gives the responder a way to examine that possibility:

100 * node_memory_MemAvailable_bytes
    / node_memory_MemTotal_bytes

Available bytes divided by total bytes, multiplied by 100, gives the percentage of memory available. These are current quantities, so this calculation does not need a rate. A fall beside the checkout failures would give the responder a lead, not establish that memory caused them. Reading the result as bytes per second would describe something else entirely.

The calm fleet response-time chart deserves the same care. A histogram counts observations in buckets, such as ranges of request duration. Check those buckets and which requests have been grouped together before calculating a percentile, a value describing a chosen position in the ordered durations. Averaging percentiles already calculated for separate regions or instances does not produce the fleet's percentile. The regional complaint needs a calculation over the relevant requests, not an average of convenient summaries.

An incident dashboard needs a route through the evidence
Arrows lead from user errors or latency to the affected traffic and population, then to same-period dependency and resource signals, and finally to logs or traces of the requests.
A suggested route through the dashboard: find the operation that is failing, narrow to the requests involved, then compare resources and inspect their logs or traces. Keep checking how recent the measurements are.
Follow the requests from symptom to investigation

The conceptual route starts with the user symptom, narrows the traffic being examined, compares dependencies and resources during the same period, and reaches individual request records. It depicts a reading order, not a measured incident. Diagram labels: User symptom: Errors or latency for the affected operation; Traffic + population: Region, tenant or other relevant scope; Dependency and resource signals: Compare against the same period; Logs or traces: Inspect the requests behind the aggregate.

When the error line disappears

The regional error line disappears. It is tempting to call that recovery, but there are other explanations: requests may have stopped arriving, or collection may have stopped. Traffic and measurement freshness help separate those possibilities. The blank panel alone cannot.

Include a missing target and a failed query in the dashboard trial, alongside a service that is genuinely quiet. The responder should be able to recognize the difference instead of having to remember which kind of blank is reassuring.

A deployment marker near the first failure is another useful lead. Check the version and region before following it: did that change reach the requests now failing? Two events being close in time helps you choose where to look; it does not establish a cause.

Then follow the checkout log link. Where supported, it should carry the service, environment and incident interval with it. If yesterday's regional failure opens as the latest fifteen minutes, healthy current requests may seem to contradict the old chart. The link has opened successfully while quietly moving the investigation to another time.

Hand the dashboard to the next person

Give another responder a known incident and its fixed start and end times. Ask them to reach the affected requests from the fleet summary, compare values and units with direct source queries, and open the matching records. Include missing data and the rotation's actual restricted permissions. A log link they cannot open breaks the same route as one that drops the time window.

Keep that route intact when the dashboard changes. Version the queries and review changes in what they count. Otherwise removing requests from a query can look like checkout improved. Choose the refresh interval with similar care: asking the source more often can add load without providing new information. The metrics guide explains counters, gauges and distributions in more detail.

Our hypothetical responder still has a cause to investigate. But now the investigation follows the failing checkouts: the wrong region's busy host has stayed out of the way, the missing line has raised a question instead of ending one, and the log link has kept the right interval. The dashboard has helped the next person pick up the incident where the customers are experiencing it.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next