On this page3 sections
Look at two panels on a hypothetical release dashboard. The one on the left says successful requests got faster. The one on the right says more customers timed out. The panels can both be right. If the duration query keeps only successes, a request vanishes from the chart as soon as it fails. The slow work is still there. The graph has lost sight of it.
Before you use a metric to make a release decision, find out whose work it counts. A number can be calculated correctly and still answer a smaller question than the one the team is asking.
Some measures keep a tally. Others keep a shape.
A request counter records how many requests have arrived since counting started. Read the increase over an interval to see how many arrived during that interval. If the process restarts and the counter begins again at zero, the drop marks a reset. A rate calculation needs to account for it instead of subtracting the old process’s work from the new one.
Now look at the queue. Its depth tells you how many jobs are waiting at that moment. A measure that can rise and fall with the current quantity is a gauge. Add the depths of two separate queues to count their waiting jobs together. If two workers are reporting the depth of one shared queue, adding their readings counts the same jobs twice.
For duration, it helps to keep some of the shape. A histogram counts how many observations fall within defined ranges, or buckets. You can see a large group of quick responses and a smaller group taking longer, even if the two produce the same average as another workload. A histogram can help estimate a percentile, the duration at or below which a given share of observations falls. The bucket boundaries and the query determine how precisely it can answer. The Prometheus guide to histograms and summaries shows why averaging each instance’s percentile cannot recover the percentile for all requests.
Even a correctly combined histogram cannot put the missing timeouts back. Keep the eligible request count, failures, missing observations and time window beside it. Now the person reading the release chart can see the difference between faster successful requests and better service for everyone who tried.
Read diagram description
The diagram compares four questions. A latency distribution shows how long measured requests took. Queue age tells you how long pending work has waited. Error fraction describes the share of eligible requests that failed. Resource usage helps identify capacity that might constrain the work.
Two ways to get 0.4 percent
One release has 400 failures in 100,000 eligible requests. Another has two in 500. Both come out to 0.4 percent. Put those counts next to the rate and the small release stops looking like an equally large observation.
The word “request” can hide another difference. Suppose someone submits an order, the first attempt fails and a retry succeeds. The service handled two attempts and the customer completed one order. Count attempts to understand the failure and extra work. Count operations to understand the eventual customer outcome. State which one the chart uses.
Now give the small group ten failures instead of two. Its rate rises to 2 percent. If the two groups cover separate requests in the same measurement interval, add the failures and the eligible requests separately. The combined rate is (400 + 10) ÷ (100,000 + 500), about 0.408 percent. The simple average of 0.4 percent and 2 percent gives 1.2 percent because it gives 500 requests the same influence as 100,000.
If a region stops reporting altogether, there are no counts from that region to add. Label the aggregate with the coverage you actually have and establish event identity or a single point of observation before treating it as a customer rate. The measurement completeness lab walks through how missing coverage changes what a result can tell you.
Follow the part of the chart that moved
Break a measurement out by release version or region and a small regression may finally show through the total. Each distinct combination of labels also creates another time series, a separate history of values. A few service versions can make a useful comparison. A raw URL or a user ID on every metric can create a new series for every distinct value. Put that finer detail in controlled logs or traces where it can help answer a particular question.
If checkout latency goes up on one version, you have somewhere to look. Check which requests reached it, when the rollout started and whether its traffic took a different route through a dependency. Carry that population and time window into a log search or trace a representative request to find where it waited. You can investigate a pattern before knowing its cause. Making it a release gate takes an additional step: explain which measured outcome should change the release decision and why.
Back at the two panels, keep the duration line and put the timeout count next to it. Now there is a question the team can act on. Which requests got faster, and what happened to the ones that never made it into the line?
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
Report an error or outdated detail