Reading time, concurrency and gaps in a distributed trace

Two dependency calls lasting 200 milliseconds do not necessarily make a request take 400 milliseconds. If they run concurrently, they can occupy the same 200 milliseconds of elapsed time. A trace waterfall is useful precisely because it preserves relationships like this, but reading only the lengths of its bars can lead the investigation astray.

A distributed trace records instrumented operations and their relationships. It helps explain where a particular request spent time and which dependency deserves a closer look. Its most valuable result is often a better next question, rather than a complete causal explanation of the system.

Concurrent children share elapsed time

A span represents an instrumented operation with timing and contextual attributes. Parent-child relationships can connect operations across services, while links can represent other relationships, including asynchronous work. Propagated context lets separate processes associate the work they record.

Begin at the customer-facing operation and follow the relevant relationships. Concurrent children overlap, so their durations cannot simply be summed into the parent's elapsed time. A long parent with few recorded children raises a different question: the remainder could include local work, waiting or missing instrumentation. A gap is evidence of unexplained time, not evidence of one specific invisible operation.

The OpenTelemetry overview introduces the common telemetry framework and its role in consistent instrumentation. The application still determines which operations are recorded. Consistent span names and attributes improve interpretation without making the record exhaustive.

Concurrent spans do not add up to elapsed time

Read diagram description

Simplified trace: a 300 ms parent contains two concurrent 200 ms calls. Together the calls occupy 200 ms of wall time, not 400 ms. The remaining 100 ms requires investigation; it is not automatically a specific missing operation.

Even an incomplete trace can locate enough of a wait to narrow the search. Its gaps remain relevant to the conclusion, but they need not prevent the responder from using the relationships and timings that were recorded.

Consider a hypothetical database span that occupies most of a slow request. One responder proposes more database capacity; another suspects locking related to a migration. The long bar locates a wait but does not choose between those mechanisms. Database state and event evidence are needed before selecting the change.

The next check is database evidence that can separate locking from insufficient capacity. The trace locates the long wait; concurrency, sampling and instrumentation gaps determine how much of its cause remains unseen. Following that lead into an independent observation is what turns the waterfall into a useful investigation.

Comparing a slow trace with similar successful work

Compare the slow execution with faster successful requests drawn from similar work. Release, region, request type and dependency route help establish whether the comparison is meaningful. One trace shows what was recorded for that execution; neighboring examples can reveal which difference deserves further investigation.

Retries can explain a long request that ultimately succeeded. Attempt timing and backoff preserve waits hidden by the final result. They also help the responder ask whether repeated attempts added dependency load. The successful endpoint status alone cannot show that sequence.

Sampling and missing spans change the available picture

Retained traces are a selected view of the service. Sampling may omit requests or spans, and a policy favoring errors intentionally preserves an unusual population. Such a collection can be excellent for diagnosis without representing the fleet's overall failure rate. Population-level metrics provide the broader comparison.

Within a trace, cross-host clock differences can complicate timing. Propagation failures and collection loss can also create incomplete trees. Investigate those paths alongside the application when time remains unexplained. Filling the picture with an assumed operation makes it look complete without adding evidence.

If the database question remains unresolved, the instrumentation guide can help extend instrumentation around the missing relationship. The investigation now has a focused purpose: determine which query, lock, resource or connection accounts for the observed wait. Reading the trace has been useful when that next check follows from what was recorded, with its limitations still intact.