Run an incident tabletop that lets people think
Give the group a delayed report, a missing approver and a backlog that outlasts the repair. Watch their decisions instead of making them guess your plot.
Practical methods you can take back to your team.
Give the group a delayed report, a missing approver and a backlog that outlasts the repair. Watch their decisions instead of making them guess your plot.
Twenty failed exports can be 0.2% of the fleet and 20% of one region. Follow the counts to see what each metric can tell you and what remains outside its view.
Three errors at the same time may belong to three different jobs. Stable fields let you follow the right export without collecting its private contents.
Two lookups overlap, so adding their durations counts some waiting twice. Read the shared timeline to find both the overlap and the time still unexplained.
Start with work you recognize: one synthetic export. Three terminals show its acceptance, background completion and recorded steps without touching a production service.
A background job can succeed while its connection to the request disappears. Break and restore one local handoff to see the difference.
Start with what arrived, what finished and what is still waiting. The machine charts can explain the delay once the customer’s work is visible.
A correct calculation can still end in a quiet inbox. Follow an export alert through its condition, route, message and recovery check.
A job ID helps you find one export. Put it on every metric and it can create a new storage problem. Keep the count and the individual story in useful places.
Send a local service’s telemetry to the wrong port, then compare its recorded job state with missing spans. Repairing telemetry delivery does not bring every old trace back.
The slow trace is useful. The command, restart and comparison that explain it are useful too. Keep them together as you follow a local export from acceptance to completion.
A receiver, processor and exporter can all be defined without forming a working route. Follow one local trace, break a reference and put the route back together.