Opens in a new tab
Explainer

Why the latency chart can improve while more customers time out

In brief

A faster latency line can lose the requests that timed out. Follow the counts, gauges and distributions back to the work they measured.

5 min read

Is this the right starting point?
Before you read
Basic service monitoring and percentages; familiarity with requests, retries and deployment versions helps apply the examples.
What you will take away
Choose suitable metric types and combine rates without hiding missing requests, population differences or unsupported percentile arithmetic.
Sources
An outer loop surrounds short rods inside a measurement frame and a long coral rod excluded from it.
On this page3 sections

Look at two panels on a hypothetical release dashboard. The one on the left says successful requests got faster. The one on the right says more customers timed out. The panels can both be right. If the duration query keeps only successes, a request vanishes from the chart as soon as it fails. The slow work is still there. The graph has lost sight of it.

Before you use a metric to make a release decision, find out whose work it counts. A number can be calculated correctly and still answer a smaller question than the one the team is asking.

Some measures keep a tally. Others keep a shape.

A request counter records how many requests have arrived since counting started. Read the increase over an interval to see how many arrived during that interval. If the process restarts and the counter begins again at zero, the drop marks a reset. A rate calculation needs to account for it instead of subtracting the old process’s work from the new one.

Now look at the queue. Its depth tells you how many jobs are waiting at that moment. A measure that can rise and fall with the current quantity is a gauge. Add the depths of two separate queues to count their waiting jobs together. If two workers are reporting the depth of one shared queue, adding their readings counts the same jobs twice.

For duration, it helps to keep some of the shape. A histogram counts how many observations fall within defined ranges, or buckets. You can see a large group of quick responses and a smaller group taking longer, even if the two produce the same average as another workload. A histogram can help estimate a percentile, the duration at or below which a given share of observations falls. The bucket boundaries and the query determine how precisely it can answer. The Prometheus guide to histograms and summaries shows why averaging each instance’s percentile cannot recover the percentile for all requests.

Even a correctly combined histogram cannot put the missing timeouts back. Keep the eligible request count, failures, missing observations and time window beside it. Now the person reading the release chart can see the difference between faster successful requests and better service for everyone who tried.

Match the measurement to the operational question
Four measurement choices matched to questions about latency, queue age, errors and resource use.
The measurement changes with the question. Keep units, included events and the time window visible so a chart answers the question you meant to ask.
Read diagram description

The diagram compares four questions. A latency distribution shows how long measured requests took. Queue age tells you how long pending work has waited. Error fraction describes the share of eligible requests that failed. Resource usage helps identify capacity that might constrain the work.

Two ways to get 0.4 percent

One release has 400 failures in 100,000 eligible requests. Another has two in 500. Both come out to 0.4 percent. Put those counts next to the rate and the small release stops looking like an equally large observation.

The word “request” can hide another difference. Suppose someone submits an order, the first attempt fails and a retry succeeds. The service handled two attempts and the customer completed one order. Count attempts to understand the failure and extra work. Count operations to understand the eventual customer outcome. State which one the chart uses.

Now give the small group ten failures instead of two. Its rate rises to 2 percent. If the two groups cover separate requests in the same measurement interval, add the failures and the eligible requests separately. The combined rate is (400 + 10) ÷ (100,000 + 500), about 0.408 percent. The simple average of 0.4 percent and 2 percent gives 1.2 percent because it gives 500 requests the same influence as 100,000.

If a region stops reporting altogether, there are no counts from that region to add. Label the aggregate with the coverage you actually have and establish event identity or a single point of observation before treating it as a customer rate. The measurement completeness lab walks through how missing coverage changes what a result can tell you.

Follow the part of the chart that moved

Break a measurement out by release version or region and a small regression may finally show through the total. Each distinct combination of labels also creates another time series, a separate history of values. A few service versions can make a useful comparison. A raw URL or a user ID on every metric can create a new series for every distinct value. Put that finer detail in controlled logs or traces where it can help answer a particular question.

If checkout latency goes up on one version, you have somewhere to look. Check which requests reached it, when the rollout started and whether its traffic took a different route through a dependency. Carry that population and time window into a log search or trace a representative request to find where it waited. You can investigate a pattern before knowing its cause. Making it a release gate takes an additional step: explain which measured outcome should change the release decision and why.

Back at the two panels, keep the duration line and put the timeout count next to it. Now there is a question the team can act on. Which requests got faster, and what happened to the ones that never made it into the line?

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next