The job finished. Why does its trace look incomplete?

The reply comes back quickly. The export finishes two seconds later. Yet the request's trace seems to stop at the reply, as if the rest of the work never happened.

There are several possible stories here. The job might not have run. It might have run without being recorded. Or its records might exist, disconnected from the request you are following. This synthetic local Dispatch exercise makes that last case visible by deliberately breaking the connection and restoring it.

Use the service and Collector from the previous lesson. The Collector receives records emitted by the service. Keep it running, and stop any earlier service process before starting the one below.

One request, two pieces of work

python3 dispatch-lab.py --worker-delay 2

Send one POST request to http://127.0.0.1:8090/exports. Save the returned job and trace IDs, then query the job's status path after completion. In normal mode, accepted_trace_id and worker_trace_id should match.

The request and worker each leave a record of their own work, called a span. Related spans are grouped in a trace. The acceptance span can finish before the worker span does: the HTTP connection has received its answer while the export continues. A two-second worker delay is not a two-second HTTP response.

At the handoff, the service gives the worker the trace ID and the accepting span's ID. The worker keeps that trace ID, creates a new span for its own work and names the accepting span as its parent. Those carried identifiers are context. Passing them across the boundary is context propagation. That connection lets us follow the work from one part to the next.

The job ID serves a different purpose: it identifies the export inside the application. It can remain useful through retries even if the tracing policy puts different attempts in different traces. The identity of a durable job and the identity of a trace need not last for the same amount of time.

Break the connection, not the export

Stop the service, then restart it in the deliberate failure mode:

python3 dispatch-lab.py --worker-delay 2 --break-context

Send a new POST and save its new identifiers. Look at the status response first. If the job succeeded, this injected fault did not prevent the export. Now compare its acceptance and worker trace IDs. They differ: this mode starts the worker's trace without the parent relationship.

Find the acceptance trace in the Collector's output, then find the worker trace separately. The worker's records may still be present. What disappeared was the route between the two sets. The job ID still connects the logs, giving you another way to investigate the same export.

Adding more spans would not necessarily answer the missing-work question. More disconnected records could leave you searching through a larger pile. The useful repair is at the handoff that was meant to connect them.

Restore the handoff and keep both results

Remove --break-context, restart the service and send another request. Keep the broken and restored results with the exact command change. Each restart resets the lab's counters and job store, so absolute counts from separate runs are not one continuous history.

Check that the request again leads to the worker's recorded work. Matching IDs establish the intended correlation, not complete coverage of every intermediate operation. Queue waiting, retries or a later download can still be absent from the trace.

A queue has to carry the connection too

In a real queue, context travels in message metadata and the consumer must read it when processing the message. A batch may combine several inputs. Links to the contributing work can represent that relationship more honestly than selecting one misleading parent.

This lab has no durable queue. It does not model separate hosts, authentication or production propagation formats. Its narrower question is whether the receiving worker got enough context to connect its work to the request that started it.

Keep the connected, broken and repaired examples together. When a trace seems to end too soon, they give you three distinct possibilities to investigate: work that did not happen, work that was not recorded and recorded work whose connection was lost. In this exercise, the export could finish while its trail broke. The repair makes that finished work findable again.

Sources

OpenTelemetry context propagation and trace concepts describe context and asynchronous relationships. The local lab is synthetic.