Leave an investigation someone else can repeat

Your colleague opens the investigation the next morning and finds a slow trace. They can see the wait, but not the command that produced it. Did the service restart? Was that the run with a two-second delay? The conclusion may be sound; the route to it has gone home with you.

Keep that route in this capstone. In the disposable local Dispatch lab, compare two synthetic runs, try another explanation for a missing span and repair a broken connection between trace records. The goal is a comparison another engineer can repeat, not a screenshot they must take on trust.

Write the command before the first request

Start the Collector and run python3 dispatch-lab.py --worker-delay 0.2. Submit three exports, one at a time, recording each ID, status, logs and spans. Stop the process and repeat with --worker-delay 2.

Name the runs baseline and delayed. Save each launch command before submitting work. A restart resets counters, so these are separate runs; do not combine their cumulative counts into one rate. Generated identifiers will differ, but the saved settings tell the next reader where each observation belongs.

One request at a time keeps the question small: does the longer configured sleep appear in the worker interval and the wait for job success? This is not a test of capacity or concurrent work. That narrower comparison makes the result easier to explain.

The quick acceptance is only the beginning

Compare the acceptance span with the worker span. The HTTP 202 response can remain quick while the worker takes approximately the configured delay. The customer’s job completes after that work, so timing acceptance alone misses the wait you introduced.

Keep the job ID, trace ID, run settings and time window beside each observation. They connect the quick response to the slower work that follows. The supported explanation is a controlled delay in this worker stage. Dispatch does no real export storage in this lab, so the trace cannot establish that a production storage service is slow.

An empty space in a trace can have several causes

A missing worker span could mean broken context, failed delivery or a client looking up the wrong job. Try at least one alternative from the earlier fault modes. With --break-context, the job still completes but the worker appears under another trace. With the wrong OTLP port, status and logs can show progress while the Collector receives no spans.

Write the prediction before checking. A longer delay should lengthen the worker interval while acceptance can stay fast. Broken context can leave an on-time completion under another trace. Failed delivery can leave the Collector empty even though status and logs show the job. Now the next observation has something to distinguish; it need not merely agree with the first explanation that sounded plausible.

Put the worker back in the same trace

For the required repair, capture a disconnected run with --break-context, remove that flag and repeat with the same worker delay. Context propagation is the handoff that carries the trace relationship into the worker. Check that the acceptance and worker trace IDs now match and both spans appear under one trace.

Save the before-and-after pair: different IDs in the broken run, matching IDs in the repaired one. Changing only the relevant flag makes the restored relationship easier to attribute. Several simultaneous changes would leave more explanations open.

The result concerns this tested handoff. It does not establish complete queue timing, production sampling coverage or correct document generation. Those are other questions, requiring observations this comparison did not provide.

Let the next engineer get there without your memory

Save the question, setup, exact commands, identifiers, observations, alternative explanation, decisive test, repair and limitations. Place raw records or repeatable capture instructions beside screenshots so the useful detail survives outside the image.

The next engineer need not reproduce your exact IDs or durations. They should be able to reproduce the qualitative behavior: find the controlled stage that caused the wait and rerun the comparison that separated another explanation. When they open the slow trace tomorrow, the command will be there too. They can begin with the investigation instead of beginning by asking where you left it.

Sources

OpenTelemetry tracing and context propagation explain the mechanisms. Every run described uses the supplied synthetic service.