Opens in a new tab
Guide

How to trace a request into its background job

In brief

Follow one background job from handoff to completion, then check whether the trace can tell waiting time apart from worker execution.

4 min read

Is this the right starting point?
Before you read
Basic application development, queues and service telemetry; access to the handler, producer, worker and tracing backend is needed for the plan.
What you will take away
Plan and verify tracing across an asynchronous handoff, separating pre-worker wait from execution and checking sampling retention.
Sources
A red ribbon winds through a theatre ticket booth, props workshop and lighting desk to a person outside the side door.
On this page3 sections

Imagine clicking Export and seeing an acceptance message before your hand leaves the mouse. The file takes much longer to arrive. The web request looks fast in the dashboard, and the worker says it finished quickly, too. Somewhere between the two, the customer was waiting.

Make that gap the job of your first distributed trace. Record when the request hands work off, when a worker starts it, and when it finishes. Then check that those records stay connected and can be retrieved with the sampling settings you intend to run. You can do that for one useful journey before instrumenting every service it touches.

This is a plan for placing and checking instrumentation. To try it, you need access to the HTTP handler, the code that sends the message and the worker that receives it. You also need tracing libraries for those runtimes and a collector or backend where you can retrieve a known job's records.

The worker needs to know which request sent the job

The web handler records its work as a span, a timed record of an operation. The handler can finish its span and send the response while the export keeps moving. Give each service a stable identity and enough deployment context to tell which version did the work. That is how you recognize a queue or worker you've instrumented when several versions are running.

When the handler sends the job, its tracing library can write identifying context into message metadata. The receiving library reads that context before it records related work. OpenTelemetry calls that context propagation. A trace ID identifies the trail and a span ID identifies a particular operation on it. A job ID is useful for finding the same job in logs, but does not by itself establish a tracing relationship. Use the library's propagator to carry the context across the boundary.

A worker taking one message can record work related to that message's creation. If it takes a batch from several requests, you may need span links, which record relationships without giving the batch several parents or pretending that all those requests made one synchronous call. Use the relevant messaging conventions and check their status and your library's support.

Suppose job-17 was handed off at 14:00:00.100, started at 14:00:00.600 and finished at 14:00:00.900. With a matched job and synchronized clocks, those readings put 500 milliseconds between handoff and start and 300 milliseconds inside the worker. The first interval can include transfer and delivery as well as time in the queue. A skewed clock or a mismatched job can spoil the subtraction.

Timeline for job-17. Handoff at 14:00:00.100, worker start at 14:00:00.600, and finish at 14:00:00.900. The intervals are 500 and 300 milliseconds.
The example puts half a second before the worker starts and three tenths inside the worker. The mobile diagram shows the order of events without a time scale.

If you're trying to explain that first half-second, a finer breakdown of the worker's execution won't help yet. A timestamp at message receipt, for example, can help separate delivery from a wait in the worker's own intake. Add the observation where it could settle the question.

Give the trace a wait you know about

Send a recognizable job through a safe environment and pull up its trace. You expect to find the request, the message handoff and the worker's record, with enough context to connect them. If the trace ends with the HTTP response, check separately whether the worker got the job and whether its instrumentation recorded and exported the relationship. The missing record can't tell you which path failed.

Now introduce a controlled delay and see whether the trail puts it where you inserted it. Try an error or retry if that behavior matters to the real investigation. Several attempts may belong to one customer's job, and concurrent operations can occupy the same time on a trace. Counting them as separate requests or adding all their durations would give you a different story from the one the customer experienced.

Make sure the evidence survives collection

Traces take space and processing. Sampling keeps some of them. An early decision can discard a request before its worker hits an unusual error. A decision made later can use more of the finished work, at the cost of buffering and processing the incoming records. Choose around the evidence you need and the capacity you have to collect it.

Rerun the delayed-job check with your intended policy. A trail that appears when everything is kept tells you the instrumentation can work. Running with normal settings tells you what the responder can expect to find. A set selected for slow or failed work also needs care when you try to generalize from it. Use metrics for a measured fleet population and traces to inspect particular executions.

Keep fields that help the responder find the job and understand its path. Raw request bodies and sensitive message contents can cost storage and expose details the responder doesn't need. Check who can retrieve the records and how long they remain available.

Return to the late export. The first project is useful when you can open a retained trail and tell which interval to investigate. Once you know the time went missing between handoff and worker start, you can choose the next measurement with a reason. The guide to reading a trace picks up when those records are already on the screen.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next