Evaluate incident-response agents against service outcomes

An incident agent can identify the likely cause, propose a sensible fix and announce recovery while requests are still failing. A review of its answer might reward all three sentences. A service check would reject the last one.

Evaluating an incident-response agent means testing the complete path from available evidence to a useful, permitted result. The model matters, but so do its tools, permissions, retrieved documents, retry behavior and the checks that decide whether work is finished. This guide develops a small evaluation you can run before giving an agent a larger role in incident response.

The immediate goal is modest: distinguish an assistant that helps an operator reach a justified next step from one that merely produces a plausible account. You will build cases, separate diagnosis from recovery, grade unknown outcomes explicitly and decide whether a pilot deserves to continue.

Choose the job before choosing the score

Start with one operational task. “Handle incidents” is too broad for a useful first evaluation. “Identify the configuration change most likely responsible for this checkout failure, cite the evidence and propose the next permitted check” gives the evaluator something concrete to inspect.

A read-only triage assistant and an agent that changes infrastructure need different acceptance criteria. The first can succeed by escalating an unresolved hypothesis with good evidence. The second needs additional proof about the affected system, including whether its action achieved the intended result. Neither should receive credit for asserting something it could not observe.

Anthropic’s agent-evaluation guidance, published January 9, 2026, distinguishes a task, repeated trials, the interaction record and the final outcome. That distinction is particularly useful for SRE: a transcript describes the interaction, while an outcome check asks what actually became true. Evaluate the deployed model-and-tool combination, rather than substituting a bare chat prompt for the system you plan to operate.

This is a timely concern as vendors put agents deeper into operational workflows. Google’s September 18 account of agents in infrastructure security is one recent example. A vendor’s description of its own deployment supplies context, not evidence that your agent can recover your service.

Build a case bank around decisions

For a first pilot, choose a handful of sanitized incidents your team understands well. Five cases with inspectable ground truth can expose more useful defects than a large collection of poorly labeled alerts. Expand the collection as you learn; the starting size is a practical suggestion, not a statistically sufficient sample.

For each case, create two packages. The agent-visible package contains only information it could have obtained at the decision point: selected telemetry, topology, configuration history and runbook excerpts. The evaluator-only package contains the expected facts, acceptable alternatives, prohibited actions and checks required to establish success. Keep the latter outside the agent’s readable workspace.

Do not give the agent the completed postmortem as its incident input. That leaks the diagnosis and retrospective explanation into a task supposedly measuring investigation. Instead, reconstruct a time-bounded evidence set and have an incident participant review whether the task remains solvable from it.

Case variation	What it tests	Evidence of a useful result
A bad configuration with a clear change record	Connecting a symptom to a plausible cause	Correct resource and supporting observations
The same symptom with stale telemetry	Recognizing limits in the available evidence	Explicit uncertainty and a fresh-data request
A proposed fix outside the allowed scope	Respecting the execution boundary	No prohibited action; an actionable escalation
A write followed by a lost response	Reconciling an unknown outcome	Destination inspection before another write
The alert clears but the user path still fails	Verifying recovery independently	Recovery remains unconfirmed

These are proposed test variations, not reports of incidents. Preserve alternative valid diagnoses where the evidence supports them. An evaluator that insists on one exact phrase can punish sound reasoning while rewarding a memorized label.

Separate the agent from its examiner

Use an isolated environment with synthetic data, a fixed starting state and credentials that cannot reach production. Give the agent the same tool descriptions and permission checks intended for deployment, but point them at fixtures or a disposable service. Reset state before every trial so one attempt cannot solve the next attempt’s incident in advance.

The evaluator should collect tool requests and results at the execution boundary. It should also inspect service state through a separate observer. Do not let the agent write the file that decides whether it recovered the service. A model-generated field such as recovered: true is a claim to grade, not the ground truth.

Record the model identifier, prompt revision, tool-schema revision, case version and environment image alongside each result. Add timeouts, concurrency and resource limits. Anthropic’s study of infrastructure noise shows why execution resources belong among the controlled variables in agent evaluations. A failed tool caused by a constrained test host should remain visible as an environment failure, rather than silently becoming evidence against a model.

Keep policy enforcement outside the grader as well. The test should block a forbidden action when attempted and record that attempt. Waiting until the final score to notice an unauthorized write would make the evaluation itself an unsafe execution environment. The site’s agent execution-contract guide explains that boundary.

Grade recovery with code before grading prose

Use deterministic checks for facts that have an observable answer: whether a prohibited action was attempted, whether the expected configuration is active or whether a synthetic request succeeds. Reserve human judgment for questions such as whether the proposed next step follows from the evidence. A model-based grader can assist with that review, but first compare its decisions with reviewers who understand the service.

The downloadable Python example and sample observation implements a deliberately small recovery gate. It consumes evaluator-owned observations, records missing evidence as unknown and refuses to let a good-looking recovery compensate for an attempted forbidden action. It uses only the standard library and does not call a model or touch infrastructure.

python3 incident-agent-grader.py --self-test
python3 incident-agent-grader.py sample-observation.json

The sample describes an illustrative rollback exercise. The observer reports successful user-path checks, no collateral change and whether the action audit is complete. The agent’s final message is deliberately absent from the gate. A single passing request is not enough for a real production recovery criterion: replace this toy observation with the service’s agreed measurement window, traffic coverage and error limits.

{
  "trial_id": "rollback-example-1",
  "audit_complete": true,
  "forbidden_action_attempted": false,
  "user_path_ok": true,
  "collateral_change": false
}

The grader returns pass, fail or unknown with reasons. Keep unknowns in the report’s denominator. Dropping incomplete trials can make an unreliable collection path look like a better agent. This example checks the grading logic only; it is not an agent benchmark or proof of production readiness.

Measure usefulness without hiding failure

Publish a small result table for each candidate configuration: valid trials, independently verified successes, policy violations, unknown outcomes, elapsed time and cost. Split results by case type. A single average can hide an agent that handles familiar configuration mistakes but fails whenever telemetry is incomplete.

Repeat each case under the same conditions and retain every attempt. If two of three trials pass, report that result rather than keeping the best answer. Small repeat counts help reveal variability, but they do not estimate rare-event risk with useful precision. Increase testing where a decision depends on a distinction the current sample cannot support.

Calculate cost per verified successful task from all costs incurred in the evaluated batch, including failed attempts and retries. As an illustrative calculation, $12 of model and tool charges across 20 trials with 12 verified successes is $1 per verified success. That figure excludes engineering and review time unless you explicitly include them; do not present it as total operating cost.

Measure the human work separately. Ask a reviewer to use the agent’s evidence to choose the next action, then record the time spent checking citations, correcting claims and retrieving missing context. An assistant that reduces drafting time but increases investigation time may still be a poor fit for the task. Our token-usage guide supplies the broader cost context.

Use public benchmarks within their scope

ITBench provides an open framework and Kubernetes-based incident scenarios for evaluating IT automation. It is a useful source of task design and environment ideas. Before adopting a scenario, inspect its prerequisites and isolate its fault injection from shared services.

The ITBench-AA methodology evaluates Kubernetes root-cause analysis from offline snapshots and scores structured diagnoses against ground truth. That is relevant evidence for diagnosis under those conditions. It does not establish that the same model can execute a live repair, handle an ambiguous write or recognize recovery in your environment.

Use public results to choose candidates worth testing, then run your local cases with your actual tools. Keep a held-out set that is not used while tuning prompts. If developers repeatedly inspect every test failure and optimize directly for it, the score increasingly measures adaptation to that collection. Add newly observed failure patterns, while preserving enough stable cases to detect regressions.

Move through a bounded pilot

Assign an evaluation owner, a service owner and someone authorized to stop the pilot. Start in offline replay. Move to read-only shadow use only after the evidence is useful: the agent observes or receives copied inputs but cannot act on the service. Compare its recommendations with the actual response without letting its output quietly become an instruction to the on-call engineer.

If a later pilot permits writes, restrict it to one reversible operation with explicit target and parameter limits. The broader principle follows Google’s canarying guidance: limit exposure, evaluate the change and make the continuation decision depend on observed behavior. An offline passing score does not remove the need to watch the live service.

Define stop conditions before the trial: an attempted out-of-scope action, a false recovery claim that reaches the operator, missing audit evidence or an unresolved write outcome. Stopping new work and undoing completed work are separate procedures. First disable new agent actions, then reconcile in-flight operations, and finally use the service’s existing rollback or compensation procedure where appropriate.

Keep incident data within the organization’s approved model and storage arrangements. Sanitize secrets and customer information before creating fixtures, minimize transcript retention and restrict who can read raw evidence. Sanitization should preserve the relationships needed for diagnosis: replacing every resource name with the same token can destroy the problem you intended to test.

Do not expand the pilot when the team cannot define success, reconstruct the permitted action or distinguish missing evidence from a negative result. Continue using the existing response process while those gaps are repaired. The next useful investment may be a reliable service check or a clearer runbook rather than a stronger model.

Bring these artifacts to the first review

A useful review can fit around one case and its records. Bring the agent-visible evidence, evaluator-only acceptance criteria, exact configuration versions, all trial results and a proposed stop procedure. Have an engineer who did not build the agent explain why a passing result deserves to pass.

If that explanation depends on trusting the agent’s own summary, improve the observer before running more trials. The evaluation earns its place when it can disagree with the agent for a reason the service owner can inspect.