Run an incident tabletop that lets people think
Give the group a delayed report, a missing approver and a backlog that outlasts the repair. Watch their decisions instead of making them guess your plot.
Guides
Walkthroughs for observability, incident response and AI-assisted operations.
Browse guides with article filtersAIOps fundamentals
Start with what AI can do for operations, and where judgment still matters.
Read AIOps fundamentalsObservability for SRE
Make metrics, logs and traces work together.
Read Observability for SREIncident management with AI
Build an incident response practice that learns.
Read Incident management with AIPractical articles
Instructions, examples and implementation notes.
The fleet looks healthy. Region B does not.
Twenty failed exports can be 0.2% of the fleet and 20% of one region. Follow the counts to see what each metric can tell you and what remains outside its view.
Useful logs keep one job in view
Three errors at the same time may belong to three different jobs. Stable fields let you follow the right export without collecting its private contents.
Why a 2.4-second request can contain 2.8 seconds of spans
Two lookups overlap, so adding their durations counts some waiting twice. Read the shared timeline to find both the overlap and the time still unexplained.
Follow one local job through its reply, logs and trace
Start with work you recognize: one synthetic export. Three terminals show its acceptance, background completion and recorded steps without touching a production service.
The job finished. Why does its trace look incomplete?
A background job can succeed while its connection to the request disappears. Break and restore one local handoff to see the difference.
Build the dashboard for the person waiting on a report
Start with what arrived, what finished and what is still waiting. The machine charts can explain the delay once the customer’s work is visible.
The alert fired. Did anyone receive it?
A correct calculation can still end in a quiet inbox. Follow an export alert through its condition, route, message and recovery check.
Where should a million job IDs live?
A job ID helps you find one export. Put it on every metric and it can create a new storage problem. Keep the count and the individual story in useful places.
The job says success. Its trace is missing.
Send a local service’s telemetry to the wrong port, then compare its recorded job state with missing spans. Repairing telemetry delivery does not bring every old trace back.
Leave an investigation someone else can repeat
The slow trace is useful. The command, restart and comparison that explain it are useful too. Keep them together as you follow a local export from acceptance to completion.
Your Collector has the parts. Are they connected?
A receiver, processor and exporter can all be defined without forming a working route. Follow one local trace, break a reference and put the route back together.