The job says success. Its trace is missing.

The job’s status says success. The collector has no trace of it. In this synthetic local lab, those two observations can both be true: the worker updates a job record, then fails to send the record of its work to the collector.

Dispatch simulates an export job by waiting briefly and setting an in-memory state. It does not generate a report, store a file or provide a download. Here, success means that the simulated job reached its success state, not that a customer received a finished export.

In this synthetic local experiment, send Dispatch’s telemetry to the wrong port, watch the job’s state change to success and then repair the telemetry destination. New traces can return while old ones remain missing. Following both outcomes makes the repair much easier to describe accurately.

Use the local Dispatch service and collector from Collect the first signals. Leave the collector running, and start a fresh service process for each experiment: Dispatch keeps its counters and job records in memory.

Send the description of the work to the wrong door

Stop the current Dispatch process and restart it with this deliberately incorrect destination:

python3 dispatch-lab.py --otlp http://127.0.0.1:4319/v1/traces

The collector listens on port 4318. This command sends to unused port 4319 instead. Submit an export, save its returned status URL and query that URL until it reports the job’s success state. Then inspect the service console and /metrics.

You should see telemetry_export_failed events and a rising dispatch_telemetry_export_failures_total counter even though the simulated job can reach its success state. The exact error depends on the host. Spans are records describing parts of the operation. Allow a few seconds for both attempts to export spans to settle before comparing counts.

The order inside the worker explains the apparent contradiction. It updates the job’s state before trying to send the span. The status endpoint can therefore report success while the console reports a failed telemetry request. Neither has to be wrong; they describe different operations.

That failure counter counts attempts to send spans, not failed lab jobs. One job produces an acceptance span and a worker span. Wait for both attempts instead of expecting one counter increment per job.

Follow the same job past each handoff

Dispatch accepted the request, its worker updated the job and its telemetry request went to the wrong address. That locates the failing step in this experiment. In a larger pipeline, the description passes through span creation, transport, collector receipt, processing, onward export and finally a backend query.

Keep the original job and trace identifiers as you follow those handoffs. A search under another service name or time range can make data that arrived look lost.

Ask whether the span was created, reached the configured address, passed through an active collector pipeline and was stored by the backend where you searched. Here it stops early, at the wrong port. Adjusting the final query cannot retrieve a span that never reached the place being queried.

Bring back new traces without inventing old ones

Restart Dispatch without the incorrect --otlp value and submit a new job. Confirm its recorded success state, matching log identifiers and collector spans. This simple service does not replay the old failed spans. What the new attempt demonstrates is restored delivery for new work.

A real collector may buffer or retry exports, but the survival of old data depends on queue size, retry limits and whether the queue persists. Read that configuration before assuming everything waited safely through an outage.

An old payload needs a surviving buffer, replay source or backend record to return. A new span cannot supply that missing history. Record when delivery resumed separately from the range still missing, so the next person knows which period can be investigated through traces and which cannot.

Ask the service a question the broken route cannot answer

When telemetry disappears, ask the service directly whether it can do the work. Where possible, monitor the collector through a route that does not depend entirely on the failing exporter. Otherwise one fault can silence both the measurements and the warning that they stopped.

Keep the result as specific as the check. In this lab, the status endpoint tells you the simulated job’s state. It does not verify a generated artifact or a customer delivery, and it cannot establish that every customer of a real service would be unaffected while telemetry was missing.

Keep the job state and missing trace in the same account

Record the job’s reported success state, last working telemetry step, incorrect destination and repair, including the old spans that did not return. “New trace delivery restored” fits the result. “All historical traces recovered” does not.

The next empty chart now has two plausible explanations: the application stopped doing its work, or the route describing that work broke. Start with the work itself, then follow its observations. In this experiment the worker updated the job, but its span went to the wrong port. After the repair, a new job’s state and its spans can both be inspected. That is the connection the experiment restores.

Sources

Collector troubleshooting and resiliency describe pipeline failures and delivery limits. The unused-port experiment is a synthetic local fault.