Follow one local job through its reply, logs and trace

Make one export request, keep its job ID and follow it through three places: the reply, the service's log and the records printed by its Collector. Because you started the job, you know which work to look for. There is no unfamiliar production incident to untangle first.

The supplied Dispatch service accepts HTTP requests and simulates background work with a short delay. An OpenTelemetry Collector receives and prints records of the service's steps. Each step's record is a span; related spans form a trace. The small exercise lets you see where those records come from before you have to investigate a missing one.

Use a disposable Ubuntu VM or another approved local lab with Python 3 and Docker Engine already available. Docker setup is outside this lesson. Check python3 --version and docker version first. Use only synthetic requests. Both the service and the Collector's published port stay on loopback, reachable on the lab machine rather than exposed as public endpoints.

Two files, one local destination

Save dispatch-lab.py and collector.yaml together in one directory. The Python program uses only the standard library. Its /health route says it is listening, /exports starts a background job and /metrics displays counters held in memory.

From that directory, start the Collector:

docker run --rm --name dispatch-collector   -p 127.0.0.1:4318:4318   -v "$PWD/collector.yaml:/etc/otelcol/config.yaml:ro"   ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.165.0

Dispatch sends records to port 4318 on the lab machine's loopback address. Docker forwards that port to 4318 inside the container. The read-only volume mapping places your configuration where the Collector expects to find it.

The configuration receives trace data over HTTP using OTLP, the OpenTelemetry Protocol. A debug exporter then prints it. This is a way to inspect the records, not a lasting trace database. The Collector listens inside its container; the port published on the lab machine remains loopback-only. Dispatch builds this small teaching payload directly, whereas normal applications commonly use instrumentation libraries.

Give the three terminals the same job to describe

In a second terminal in the same lab environment, start the service:

python3 dispatch-lab.py --worker-delay 2

In a third terminal, request an export and inspect the counters:

curl -i -sS -X POST http://127.0.0.1:8090/exports
curl -sS http://127.0.0.1:8090/metrics

The POST returns a job ID, a trace ID and a status path. Substitute that job ID into curl -sS http://127.0.0.1:8090/exports/JOB_ID and query the exact path. It may first show pending. After the simulated two-second work, it should show success.

In the service terminal, look for structured acceptance, worker-start and worker-finish events. In the Collector terminal, look for spans named POST /exports and build export, with service name dispatch-lab. You are following the same job from its request to its background work, not comparing unrelated lines that happen to appear together.

An accepted request is only the first answer

The initial HTTP 202 means the request was accepted for processing. The later status says whether the job finished. Check that success, the increased completion counter and the matching trace at the Collector separately. Acceptance does not prove trace delivery; a printed span does not prove that a real customer's report is correct.

Use the job ID to follow application events and the trace ID to follow recorded steps. If the Collector stays quiet, check the service for telemetry_export_failed, then examine the Collector address and receiver pipeline. The failure counter on /metrics offers another view of export errors, outside the trace path that may itself be broken.

The two-second delay makes the handoff visible. The HTTP reply has already returned while the worker continues. You are not simply making one HTTP response take longer.

Keep the trail before stopping the lab

Save the IDs, matching events, spans and counter values. Mark any step you did not execute: reviewing a configuration is not an end-to-end test.

Stop the service with Ctrl+C, then stop the Collector in its terminal. Removing the container leaves the two local files in place. Restarting Python resets the jobs and counters; outstanding jobs do not survive. This is a private instrumentation exercise, not a durable production queue or a secure public server.

If you ran the whole path, you now have a useful reference for later fault exercises: one request, one job and a recognizable trail through the three terminals. When a piece goes missing, you can compare it with a path you have actually seen work.

Sources

Collector installation and configuration document the receiver and debug pipeline. The Python service is supplied specifically for this course.