Observability
Prometheus missing data: test what your error-rate alert cannot see
A zero fallback can hide missing telemetry. Use a small PromQL test matrix to check what your error-rate alert can and cannot establish.
Practical guidance for SRE and platform teams operating reliable services and production AI. Independent analysis, field guides, and working resources.
Selected reading
Timed-out automation writes: reconcile before retrying
Keep unknown write results visible, preserve operation identity and rehearse a lost response with a local Python fixture.
Keep an uncertain write owned until the destination can resolve it.
Read the article
Original writing / Newest first
More from AIOpsSRE
How-To
Disruption budgets and the capacity left behind
Plan an inference-service node drain with a scoped disruption budget, replacement-capacity checks and a practical maintenance worksheet.
How-To
Prompt caching: measure the cost of useful work
Build a prompt-caching pilot that counts retries, failed work and unknown charges, with a local calculator and sample ledger.
How-To
How to evaluate incident-response agents
Build repeatable cases and check recovery independently, with a runnable Python grader.
Technology Overviews
Supervising coding agents with Herdr
Check agent status, terminal persistence, and recovery before adopting Herdr.
AIOps
What MTTD misses
Separate detection from response and find delays that an improving average can conceal.
Releases, research & operational impact
Latest AI news
AI data infrastructureNews analysis
Storage Intelligence adds findings and safer batch previews
Google’s advisor and batch updates connect storage findings to bulk actions. Review the workload, selected objects and recovery boundary before execution.
Read the analysis
Agent evaluationNews analysis
Google PageBreak separates agent findings from proof
Google’s internal security project validates agent hypotheses before routing reports. Its evidence path is useful; incomplete coverage still needs an owner.
Agent infrastructureNews analysis
AlloyDB previews separate compute for agent queries
Google’s read-only agent compute preview separates query execution from production compute. Evaluate bursts, return from idle and data permissions.
From reading to doing
Work through it, step by step.
Incident response
Build a usable runbook
Work through a database connection example, from the first symptom to a verified recovery.
Worked example + Markdown templateRead the guide
Reliability planning
Put your error budget to work
Define the service objective, calculate the allowance, and agree when a release needs to wait.
Calculations + policy fieldsRead the guide
Observability
Trace a request across services
Start with one important request path. Check propagation, sampling, and the gaps in your evidence.
Implementation guideRead the guide