Read the work behind an observability metric

The release dashboard looks faster. Customers are timing out more often. In this hypothetical service, the latency panel includes only successful requests, so its slowest members disappear when they time out. The remaining requests really are faster; that is simply not the improvement the team wanted.

Metrics make a busy service small enough to think about. They keep selected counts, quantities and distributions while leaving other details behind. Reading them well means finding out which work survived that compression.

Choose the kind of measure that answers the question

A counter accumulates events, such as requests or failures. Its change over an interval tells you how many arrived. A process restart can reset a counter, so a rate calculation needs to recognize that reset instead of interpreting it as negative traffic.

A gauge records a current quantity, such as the jobs waiting now. Add queue depths when they represent separate work. Two workers reporting views of the same queue do not give you two queues to add together.

A distribution preserves variation that one average hides. Imagine grouping request durations into ranges: many quick requests and a smaller slow group remain distinguishable. Histograms support threshold and percentile questions with precision that depends on their buckets and query.

Prometheus explains the aggregation differences between histograms and summaries. In particular, averaging percentiles from separate instances does not produce the fleet percentile.

But correcting that calculation would not restore the opening example’s missing timeouts. Put eligible request counts, failures, missing work and the observation interval beside the duration distribution. Now the chart can help explain whether the service improved or merely stopped including its slowest requests.

Match the measurement to the operational question

Read diagram description

Conceptual comparison. Keep units, eligible population and collection window with each metric; choosing the convenient graph can answer the wrong question. Diagram labels: Latency distribution: How long did requests take?; Queue age: How long has pending work waited?; Error fraction: What share of eligible requests failed?; Resource usage: Which capacity might constrain the work?.

Keep the counts beside the percentage

One release has 400 failures among 100,000 eligible requests. Another has two among 500. Both report 0.4%, but the second conclusion rests on far less observed work. Show the counts so a reader can see that difference.

Also state what a request means. One customer operation might fail on its first attempt and succeed on its second. Attempt metrics describe the failure, success and extra work. A transaction metric can describe the eventual outcome of the one customer operation. Neither replaces the other.

To combine disjoint populations, add failed events and eligible events separately, then divide. If the smaller group instead has ten failures among 500 requests, its rate is 2%. The combined result is (400 + 10) ÷ (100,000 + 500), about 0.408%. Averaging 0.4% and 2% yields 1.2% because it gives the small group half the influence.

That arithmetic still requires complete, nonoverlapping counts. Establish event identity or a single observation boundary before calling the sum a customer-level rate. An unavailable population is not a population with zero failures; the measurement-completeness lab works through that distinction.

Use the summary to find the relevant detail

A region or version label can expose a local regression hidden in the total. Each distinct label combination also creates another time series, so unbounded identifiers such as user IDs and raw URLs belong in more carefully controlled event detail rather than automatically on every metric.

Suppose checkout latency rises on one version. That narrows the investigation but does not establish the cause. Its requests may have a different mix or reach a dependency differently. Compare those conditions with rollout time, then follow a representative request through targeted tracing.

Carry the affected population and interval into the log or trace link. The metric shows where to look; the detailed record helps explain what happened. Another wall of panels is less useful than one route that preserves the question.

Exploration can begin without a planned action. Before using a pattern to trigger a consequential change, however, understand its population, exclusions and relationship to that action. An interesting correlation should not silently become a release policy.

For the opening release, putting timeouts back into the account changes the question to “What happened to all the requests?” Counts describe their outcomes and distributions describe the durations that were measured. Together they can reveal the regression the flattering line concealed.