Plan your first distributed trace across a background job

The job was accepted quickly, but its result arrived late. In this illustrative service, did the job wait for a worker, or did the worker take a long time? A trace of only the HTTP request ends too soon to answer. Start the tracing project by following that one handoff.

A distributed trace connects records of work across services. The useful part is keeping that trail through the boundary you want to understand. One request followed into its background job can explain more than hundreds of disconnected operation records.

Our job has two relevant clocks. The user waits from submission until a result is ready; the worker runs only after it takes the job. Measuring the worker alone can make fast processing look like fast service while the queue holds most of the wait.

This is an instrumentation plan, not an SDK installation tutorial. Arrange access to the handler, message producer and worker, tracing libraries for their runtimes, and a collector or backend where you can retrieve the result. Then you can test whether the intended trail actually appears.

The queue handoff connects acceptance to completion

Begin at the incoming request and identify the calls and queue handoffs needed to explain its wait. The timed operation records are spans. Give them stable service identity and deployment context so the responder can connect the work with the version and configuration that performed it.

Background work needs an explicit relationship to its originating request; it is not automatically one long synchronous call. OpenTelemetry's context guidance describes passing identifying context across processes. Without that passage, a producer and consumer can both emit spans that appear unrelated.

The producer injects that context into a carrier, usually message metadata, and the consumer extracts it before recording related work. A trace ID names the trail; a span ID names one operation. A job ID can help someone search logs but does not automatically supply the tracing relationship. Use the library's propagator rather than hand-built headers.

Sometimes a span link fits better than a single parent. A batch may combine messages from several requests; links can preserve those origins without inventing one synchronous chain. Follow the relevant messaging conventions, checking their status and your library's support.

Look for the handoff time, worker start and worker finish. Together they separate the pre-worker wait from execution. Add more spans where they explain another gap, not merely to fill the display.

In a constructed trace, job-17 is handed off at 14:00:00.100, starts in the worker at 14:00:00.600 and finishes at 14:00:00.900. With matched context and synchronized clocks, that gives 500 milliseconds before worker start and 300 during execution. The first includes handoff and delivery, not necessarily 500 milliseconds in the broker alone. Clock skew or missing relationships weaken that attribution.

Read it in order: handoff, half a second before processing, then three tenths of a second running. The first half-second is the next interval to explain. More detail inside the worker might help later, but it cannot account for this earlier wait.

The first trail will not capture every important interaction. Expand when the evidence reveals an omitted dependency. A focused beginning is useful; one successful journey is not proof of complete service coverage.

Following a known delayed job through the trace

Send a known request through a safe environment and compare the expected path with the retained trace. Unexpected service names or missing records identify places to inspect. Follow errors and retries too, keeping repeated attempts recognizable as related work rather than extra customer operations.

Then introduce a controlled slow dependency. Can the trace locate the wait? Remember that overlapping child spans cannot simply be added to obtain elapsed time, and uninstrumented work can leave gaps. Use those gaps to choose the next check.

If the trace ends at the successful HTTP response, it has not explained the late job. Ask whether the job reached a worker and whether the tracing relationship survived long enough to be retained. A missing span alone cannot choose between those failures. Once you can follow the job, ask whether normal sampling will keep that trail.

Sampling trades retained evidence for collection cost

Sampling saves storage by discarding some traces. An early choice can drop a request before its rare downstream error occurs. Waiting for more of the trace permits a later choice but needs buffering and capacity. Choose for the delays and failures the team needs to investigate.

A collection selected for slow or failed requests cannot directly supply a fleet error rate without accounting for that selection. Use metrics for population measurements and traces for execution detail. Repeat the delayed-job test with the sampling policy you plan to run; a complete trail obtained only with sampling disabled does not establish normal retention.

Attributes also cost storage and need access controls. Keep identifiers that help follow work, avoid sensitive payloads and prevent unbounded values accumulating merely because a span can accept them. Retention and access decide whether the intended responder can use this evidence.

The trace-reading guide develops interpretation. For this first project, repeat the late job, find its handoff and worker records, and explain the remaining gaps. You then have a working trail and a reason to choose where the next measurement belongs.