Choosing tools for an SRE workflow

An incident moves through more than one tool: a measurement becomes an alert, an alert reaches a responder, and an investigation leads to a change. Choosing a stack means understanding those connections as well as the individual products. If a dashboard cannot identify the service named in a page, responders have to reconstruct the relationship themselves.

This guide follows that work from instrumentation through delivery. Its examples and selection criteria draw on the linked documentation and AIOpsSRE guides; they are not a ranked product comparison or a report of hands-on testing.

Instrumentation and investigation

Consider an illustrative checkout service that has become slow. The responder needs to see which operation slowed down and when, then follow an affected request far enough to understand where it waited. Measurements, event records and request traces provide those clues. Adding the code that produces them is instrumentation; the resulting data is telemetry. OpenTelemetry supplies instrumentation and collection components, but the data still needs somewhere to live and a way to search it. Buying or installing a collector alone does not supply a complete monitoring service's storage, retention and investigation features. The OpenTelemetry architecture guide follows those connections.

Now imagine wanting to compare today's checkout delay with yesterday's. The storage system must have kept yesterday's data, and its query features must let you ask that question. That is the practical difference between retention, how long data remains available, and query behavior, what you can retrieve or calculate from it. The Grafana dashboard guide follows the investigation view; the metrics guide explains which requests and measurements a chart represents.

Choosing how to label those measurements affects what the storage system must hold. Splitting traffic by service and region can help locate the delay. Add a unique request identifier as a metric label, though, and every request can create its own time series, a separate sequence of measurements. Cardinality counts the distinct values or combinations represented in the data. The costs and limits depend on the backend, the system storing and serving that data. Keep collection loss and stale data visible too: an empty panel should not persuade a responder that the problem has disappeared.

Paging and incident coordination

The next connection is between a detected condition and someone able to respond. Scheduling, escalation and acknowledgment decide who receives the page and what happens if they cannot take it. The incident record then holds the impact, current actions and next decision as people move between monitoring, tickets and chat. The PagerDuty, Jira and Slack integration guide examines where those records can diverge and how delayed notifications or unavailable chat affect the response.

For existing Opsgenie users, replacement planning has a concrete deadline. Atlassian states that Opsgenie will shut down on April 5, 2027. Migration work therefore includes schedules, routing and permissions, together with evidence that the replacement can reach the intended responder.

AI assistance and execution

Suppose, in this hypothetical checkout investigation, an assistant receives a current request trace and an old incident note. The trace shows time spent waiting for a database. The old note describes a payment-provider problem. A useful account keeps those two observations distinct and helps the responder decide what to investigate next. A polished summary that quietly treats the old explanation as today's cause leaves someone else to untangle it.

Giving the assistant a larger context window lets it receive more material at once, which can help assemble an investigation scattered across records. It can also bring in more irrelevant or conflicting material, like that old incident note. Compare whether the assistant finds and correctly explains the detail that changes the investigation, including what it leaves out and how long a responder spends correcting it. The Claude Opus evaluation article considers model configurations on operational work, while the NotebookLM guide follows a source-backed incident dossier.

The assistant is also another dependency. What happens to the investigation if it is unavailable, and where does the incident data go while it works? Those data-handling and failure behaviors belong in the comparison. Asking it to generate a patch or execute a recovery procedure adds consequences beyond preparing an account. The automated-remediation guide covers the limits on actions and the checks that establish whether recovery actually happened.

Delivery and failure testing

Eventually the checkout team may choose a change. Deployment tools take the intended configuration or software version and apply it to the running service. But a tool can finish its own step while the service remains awkward to recover. A partial rollout leaves different instances on different versions. Drift means the running setup no longer matches the declared one. An incompatible data change can leave the old software unable to work with data the new version has written.

For a hypothetical checkout change that has already rewritten stored data, turning off the new feature may stop further use without restoring what was written. Rolling software back to an older version is a different action, and that version must still be compatible with the data. The canary deployment guide explains how to select the traffic that first receives a change, decide whether to expand the rollout and check rollback compatibility. The recovery option has to fit the effect of the change, not just the buttons the deployment tool offers.

A controlled failure test can examine an assumption about that recovery path. Before introducing a fault, the team needs to say what it expects to happen, which part of the system the test may affect, what it will watch and what will stop the test. The tool supplies a way to introduce the failure; it does not choose a safe scope on the team's behalf. The AIOps evaluation guide brings these questions into a comparison with the existing workflow, including the integration and maintenance effort.

Follow one checkout failure through the proposed stack before judging its breadth. Can the alert open the relevant service view? Can the responder reach the matching request evidence? Does the incident record show the change and its result? Those connections determine how much work the team must reconstruct when the individual tools are all technically working.