Your Collector has the parts. Are they connected?
A receiver, processor and exporter can all be defined without forming a working route. Follow one local trace, break a reference and put the route back together.
Articles
Guides, news and explanations about AI and reliable systems.
Not sure where to begin? Choose a short reading path, or browse at your own pace.
A useful page tells the next responder where to begin
A growing queue is a clue, not a complete message. Help the next responder see who is waiting, what to compare first and how to recognize recovery.
The completed-job count dropped after a restart
A restart resets the lab’s in-memory counters. Follow that reset through a Prometheus query, then build panels that distinguish failed simulated jobs, inactivity and missing measurements.
Reduce toil without making the wrong retry faster
Forty retries a week look ready to automate. Follow the check before the click, then see whether a tool would remove work or simply move it.
What did those hundred canary exports actually test?
A new worker can finish a hundred reports without touching the code you changed. Choose work that answers the release question, then plan how to stop and recover.
Leave an investigation someone else can repeat
The slow trace is useful. The command, restart and comparison that explain it are useful too. Keep them together as you follow a local export from acceptance to completion.
The job finished. Why does its trace look incomplete?
A background job can succeed while its connection to the request disappears. Break and restore one local handoff to see the difference.
An unfamiliar account appears during the incident
It might be a backup job. It might need urgent attention. Bring the observation to a specialist without losing the service investigation or spreading sensitive records.
SRE, DevOps and platform teams: who gets the report to the customer?
A healthy queue and a submitted job can coexist with a missing report. Follow the handoffs to see which team can make the next decision.
The job says success. Its trace is missing.
Send a local service’s telemetry to the wrong port, then compare its recorded job state with missing spans. Repairing telemetry delivery does not bring every old trace back.
Where should a million job IDs live?
A job ID helps you find one export. Put it on every metric and it can create a new storage problem. Keep the count and the individual story in useful places.
Why a 2.4-second request can contain 2.8 seconds of spans
Two lookups overlap, so adding their durations counts some waiting twice. Read the shared timeline to find both the overlap and the time still unexplained.