Read a distributed trace without mistaking it for the whole system

A trace should support the next discriminating decision without pretending to explain unobserved work. The incident loop needs evidence that separates plausible causes and later verifies the chosen action. A long span identifies a boundary worth investigating; it does not automatically authorize a change to the component behind it.

A distributed trace records instrumented work associated with a request or operation. It helps locate waiting and failed dependency calls, while leaving the investigator to establish what caused them.

Reading a trace well means understanding both its structure and the evidence it leaves out.

Read relationships before durations

A span represents an instrumented operation with timing and contextual attributes. Related spans can form a parent-child structure across services, while links can represent other relationships such as asynchronous work. Propagated context lets separate processes associate their work with the same trace.

Start with the customer-facing operation, then follow the dependency boundaries. Check whether child operations overlap. Adding their durations can overstate elapsed time when calls run concurrently. A long parent span with few children may indicate local work, waiting, or missing instrumentation.

The OpenTelemetry overview introduces the common telemetry framework. Its conventions can improve consistency, but the application still determines which operations are recorded.

Consider a hypothetical database span consuming most of a request’s elapsed time. One responder proposes scaling the database, while another suspects locking from a recent migration. Inspect the dependency’s state before choosing the action. The trace narrows the question, but neither its duration nor its visual prominence resolves the competing mechanisms.

Concurrent spans do not add up to elapsed time

Read diagram description

Simplified trace: a 300 ms parent contains two concurrent 200 ms calls. Together the calls occupy 200 ms of wall time, not 400 ms. The remaining 100 ms requires investigation; it is not automatically a specific missing operation.

Use the trace to select the next independent check, retaining concurrency, sampling, and missing instrumentation as limits on the conclusion.

An incomplete trace can still narrow an investigation. Do not demand perfect coverage before using it, but do not let a clear-looking waterfall imply certainty about an unobserved cause.

Compare more than one example

One slow trace shows one observed execution. Compare it with successful and faster requests from the same relevant population. Check release, region, request type, and dependency route before treating differences as causal.

Retries deserve particular attention. A request may eventually succeed after several failed attempts, leaving customer latency high while the final response looks healthy. Inspect attempt timing and backoff, and determine whether retries amplified dependency load.

Account for missing and selected evidence

Sampling means some requests or spans may be absent. A dataset deliberately retaining errors is useful for investigation but unsuitable for a naive estimate of the overall error rate. Join the trace investigation with population-level metrics.

Clock differences can complicate cross-host timing. Instrumentation gaps, propagation failures, and collection loss can produce incomplete trees. Treat unexplained gaps as questions rather than filling them with a confident story.

Follow a suspected dependency problem into its own resource and event evidence. A slow database span identifies observed waiting at that boundary; it does not alone distinguish an inefficient query, locking, saturation, or network delay.

A trace is most valuable when it narrows the next check. If the current instrumentation cannot answer that check, the instrumentation guide offers a focused way to extend it.