Reduce toil without making the wrong retry faster
Forty retries a week look ready to automate. Follow the check before the click, then see whether a tool would remove work or simply move it.
Articles
Guides, news and explanations about AI and reliable systems.
Not sure where to begin? Choose a short reading path, or browse at your own pace.
What did those hundred canary exports actually test?
A new worker can finish a hundred reports without touching the code you changed. Choose work that answers the release question, then plan how to stop and recover.
Leave an investigation someone else can repeat
The slow trace is useful. The command, restart and comparison that explain it are useful too. Keep them together as you follow a local export from acceptance to completion.
The job finished. Why does its trace look incomplete?
A background job can succeed while its connection to the request disappears. Break and restore one local handoff to see the difference.
An unfamiliar account appears during the incident
It might be a backup job. It might need urgent attention. Bring the observation to a specialist without losing the service investigation or spreading sensitive records.
SRE, DevOps and platform teams: who gets the report to the customer?
A healthy queue and a submitted job can coexist with a missing report. Follow the handoffs to see which team can make the next decision.
The job says success. Its trace is missing.
Send a local service’s telemetry to the wrong port, then compare its recorded job state with missing spans. Repairing telemetry delivery does not bring every old trace back.
Where should a million job IDs live?
A job ID helps you find one export. Put it on every metric and it can create a new storage problem. Keep the count and the individual story in useful places.
Why a 2.4-second request can contain 2.8 seconds of spans
Two lookups overlap, so adding their durations counts some waiting twice. Read the shared timeline to find both the overlap and the time still unexplained.
Make the first on-call shift workable
A runbook can be correct and still stop at a login screen. Rehearse one delayed export to find the access, decisions and people a new responder will need.
T3 Code: keeping track of the agent, the branch and the actual fix
A promising answer is easier to review when you can find the files behind it. Follow one small repair through T3 Code’s sessions, worktrees and host machines.
Give the incident a clear next decision
Everyone is busy, yet the rollback is going nowhere. Assign coordination, technical work and communication without confusing a role with permission to change production.