Choosing an SLO: what the extra nine asks you to change
At a million requests, moving from 99% to 99.9% allows 9,000 fewer missed reports. Choose the reliability work and customer promise behind that number.
Articles
Guides, news and explanations about AI and reliable systems.
Not sure where to begin? Choose a short reading path, or browse at your own pace.
Fewer requests arrived. Did the mitigation work?
A submission limit can do exactly what you asked and still leave reports waiting. Compare it with a rollback, then follow the work after the change.
When storage recovers but the reports are still waiting
A five-minute slowdown can leave ten more minutes of work in the queue. Follow the exports to see how retries, limits and recovery affect the wait.
Build the dashboard for the person waiting on a report
Start with what arrived, what finished and what is still waiting. The machine charts can explain the delay once the customer’s work is visible.
Choosing the reliability work that earns the next two weeks
Retry repairs, extra workers and less manual checking all help in different ways. A useful investment proposal makes the tradeoff visible, including what would change your mind.
Error budgets: the same balance can call for a different decision
Forty exports of margin looks different after a repaired fault than during a growing failure. Connect the count, its pace and the recovery work to release decisions.
Evaluating AI answers: when a convincing fix is the wrong fix
An assistant can sound helpful while reversing the source. A small fictional export service makes the difference visible, one answer at a time.
Run an incident tabletop that lets people think
Give the group a delayed report, a missing approver and a backlog that outlasts the repair. Watch their decisions instead of making them guess your plot.
When the first mitigation helps but does not fix the incident
The request graph falls, yet the oldest report still waits. Keep the protection that worked while finding the next question and the person authorized to act.
After the incident: change what the next retry does
A report can be finished even when its caller sees a timeout. Follow one lost reply from duplicate work to a repair another engineer can test.
What to tell customers while the answer is changing
At 14:15, one region has delayed reports. Eight minutes later, you know about another. Keep people informed without promising a fix you cannot yet see.
Hand over the incident, not another mystery
The incoming lead needs the next decision, the controls still active and the changes that never ran. Follow one fictional handoff from Maya to Rowan.