Testing an incident-response agent against the service it must recover

In a hypothetical incident exercise, the agent identifies the bad configuration, describes a sensible rollback and announces that checkout has recovered. The evaluator sends a checkout request. It still fails. The explanation may contain useful investigation, but the exercise has just exposed the most important disagreement in the trial: what the agent says happened and what the service actually does.

This guide builds an evaluation capable of preserving that disagreement. The object being tested is the whole working system: model, retrieved evidence, tools, permissions, retries and completion checks. A strong answer from the model is one part of it. The trial also needs to establish whether the next step was justified, permitted and useful, and whether a claimed recovery can survive an independent check.

Keeping that ambition small at first makes the result easier to understand. One well-specified operational job can reveal why a trial passed, failed or remained unresolved. From there, the team can decide what deserves more testing and whether any additional role in incident response is justified.

What job was the agent supposed to finish?

“Handle incidents” leaves too much room for the evaluator to invent success afterward. A more useful first assignment is to identify the configuration change most likely to explain a checkout failure, cite the observations and propose the next permitted check. The expected result is then something another engineer can inspect, rather than a confident tone or a long transcript.

The distinction changes with the agent’s authority. A read-only triage assistant may complete its job by identifying uncertainty and escalating a supported hypothesis. An agent permitted to change infrastructure has a further obligation: evidence about the affected system after its action. An inability to observe recovery should limit the claim it makes, not encourage a guess about how the command probably went.

Anthropic’s January 9, 2026 evaluation guidance separates the task, repeated trials, interaction record and final outcome. That vocabulary helps here because the record of a rollback is different from evidence of restored checkout. Testing the deployed model-and-tool combination preserves that distinction; a bare chat prompt cannot stand in for tools and permissions it never exercised.

Vendors are moving agents into more operational work, including Google’s September 18 account of infrastructure-security agents. Such reports explain why the question matters. Their deployment experience does not establish what a different agent can do to your service, which is the purpose of the local exercise.

Separate evidence for the agent and the evaluator

A handful of sanitized, well-understood incidents is a reasonable place to start. Five cases with independently checkable facts may reveal more useful defects than a large pile of poorly labeled alerts. Five is a practical suggestion, not a sample large enough to support broad reliability claims. The collection should grow wherever the eventual decision depends on a distinction the current cases cannot resolve.

Each case has two packets. The agent receives the measurements, logs, dependency information, configuration history and runbook excerpts available at the decision point. The evaluator keeps the expected facts, acceptable alternatives, prohibited actions and success checks outside the agent’s readable workspace. That separation makes the exercise an investigation rather than a test of whether the answer file can be found.

A completed postmortem belongs on the evaluator’s side. Giving it to the agent would introduce the diagnosis and retrospective explanation into the supposed starting evidence. Reconstruct a time-bounded input instead, then ask an incident participant whether the task is still solvable from it. This catches an easy mistake: removing the answer so thoroughly that the evidence needed to infer it disappears too.

Case variation	What it tests	Evidence of a useful result
A bad configuration with a clear change record	Connecting a symptom to a plausible cause	Correct resource and supporting observations
The same symptom with stale telemetry	Recognizing limits in the available evidence	Explicit uncertainty and a fresh-data request
A proposed fix outside the allowed scope	Respecting the execution boundary	No prohibited action; an actionable escalation
A write followed by a lost response	Reconciling an unknown outcome	Destination inspection before another write
The alert clears but the user path still fails	Verifying recovery independently	Recovery remains unconfirmed

The variations above are proposed tests, not incident reports. They change what a useful response looks like without changing the need for evidence. Where more than one diagnosis fits, the evaluator should preserve those valid alternatives. Insisting on an exact phrase can reward a memorized answer while rejecting a sound explanation.

Who is allowed to declare recovery?

The checkout exercise needs a disposable service or fixtures in an isolated environment, synthetic data and credentials unable to reach production. It should still use the intended tool descriptions and permission checks, so the test reaches the controls the deployed agent would meet. Resetting the starting state before each trial prevents one successful repair from silently solving the next trial’s incident.

An agent’s recovery claim needs service evidence collected independently of the agent. Tool requests and results can be captured at the execution boundary, while a separate observer inspects the service. A model-produced recovered: true field is something to grade, not something that should overwrite the grader’s answer. The agent therefore must not be able to edit the observation file that establishes success.

The environment itself can change the result. Record the model identifier, prompt, tool-schema and case revisions, environment image, timeouts, concurrency and resource limits with each trial. Anthropic’s study of infrastructure noise explains why execution resources belong among controlled variables. A tool failure caused by a constrained host should stay visible as that kind of failure; quietly relabeling it as poor diagnosis would send improvement work to the wrong place.

The grader is also too late a place to enforce permission. A prohibited write should be blocked when attempted and retained in the record. Discovering it only in the final score would mean the test allowed the very action it was meant to contain. The agent execution-contract guide explains how that execution boundary fits into a capability.

A Python grader for independent service observations

Some questions have answers a deterministic check can establish: was a prohibited action attempted, is the expected configuration active, and did a synthetic user request succeed? Other questions need judgment, such as whether the proposed next step follows from incomplete evidence. A model-based grader can assist with the latter, but its decisions should first be compared with those of reviewers who know the service.

The downloadable Python example and sample observation make the first kind of verdict deliberately small. Using only the standard library, the grader reads evaluator-supplied observations, keeps missing evidence unknown and refuses to let apparent recovery compensate for an attempted forbidden action. It does not invoke a model or operate infrastructure.

python3 incident-agent-grader.py --self-test
python3 incident-agent-grader.py sample-observation.json

The sample is an illustrative rollback exercise. A separate observer checks the user request path, collateral changes and the completeness of the action record. Those findings become the JSON fields below. There is no field for how convincing the agent’s final message sounded.

{
  "trial_id": "rollback-example-1",
  "audit_complete": true,
  "forbidden_action_attempted": false,
  "user_path_ok": true,
  "collateral_change": false
}

A single successful request cannot establish production recovery; a real evaluation needs the service’s agreed measurement window, traffic coverage and error limits in place of this toy observation. The small example makes the ownership of evidence visible, without pretending that its few fields settle every service’s acceptance criterion.

The possible results are pass, fail and unknown, accompanied by reasons. Unknown trials remain in the reported total because missing observations say something important about the evaluation. Removing them improves a percentage without improving the agent or the service check. This code demonstrates grading logic, rather than a benchmark result or a production-readiness certificate.

Failures, unknown outcomes and cost per success

A result table should distinguish valid trials, independently verified successes, policy violations, unknown outcomes, elapsed time and cost for each configuration. Break it down by case type. An aggregate that combines clear configuration faults with stale-telemetry cases can conceal an agent that succeeds only when the answer is easy to see.

Repeat cases under the same conditions and preserve every attempt. Two passes out of three is evidence of variability, not an invitation to retain the best response. Small repeat counts help discover that variability but cannot estimate rare-event risk precisely. More testing is useful when it resolves an uncertainty that matters to the proposed pilot.

Task cost follows the same rule. In an illustrative batch with $12 of model and tool charges, 20 trials and 12 verified successes, the cost is $1 per verified success. Failed attempts and retries remain in the $12. Engineering and review time are excluded unless explicitly measured and included, so this figure alone is not total operating cost.

The reviewer’s work deserves its own observation. Have someone use the agent’s evidence to choose a next action, then record time spent checking citations, repairing claims and finding missing context. Faster drafting can coexist with slower investigation. The token-usage guide develops the broader accounting, while this exercise asks whether the agent’s contribution makes the actual decision easier.

What ITBench contributes to a local evaluation

ITBench offers an open framework and Kubernetes incident scenarios for IT automation. Its task design and environments can be useful starting points. Adopting one still requires inspecting prerequisites and isolating fault injection from shared services; a benchmark scenario is an executable environment, not merely a questionnaire.

The ITBench-AA methodology evaluates Kubernetes root-cause analysis from offline snapshots, comparing structured diagnoses with ground truth. That speaks to diagnosis under those conditions. A live repair, a write with a lost response and recognition of actual recovery introduce questions the offline result does not answer.

Public results can therefore narrow the candidate list before local cases exercise actual tools. Keep some cases held out from prompt tuning. Repeatedly inspecting every failure and optimizing directly for it makes the score increasingly about adaptation to that collection. Newly observed failure patterns can expand the bank, while a stable subset still helps reveal regressions.

From offline replay to a limited live role

Offline replay is the first opportunity to learn without affecting a live service. Name who maintains the evaluation, who owns the service and who can stop the trial. If replay produces useful results, read-only shadow use can compare recommendations with actual incident response while the agent remains unable to act. Those recommendations need to remain clearly distinguishable from instructions to the on-call engineer.

A later write-enabled pilot should narrow the action to one reversible operation with explicit targets and parameter limits. Google’s canarying guidance supplies the relevant principle: limit exposure and let observations determine whether the change continues. The live service still needs watching even after an offline score has earned a trial.

Stopping is a workflow to prepare, not just a threshold to announce. An attempted out-of-scope action, a false recovery claim reaching the operator, absent audit evidence or an unresolved write can each trigger it. First stop new actions, then reconcile operations already in flight, then use the established rollback or compensation procedure where appropriate. Disabling the agent alone does not reverse changes already made.

The case data also needs the organization’s approved model and storage arrangements. Sanitize secrets and customer information, minimize transcript retention and restrict raw-evidence access. Sanitization must preserve relationships needed for diagnosis: replacing every resource name with one identical token can destroy the distinction the case was designed to test.

If the team cannot define success, reconstruct the permitted action or distinguish absent evidence from a negative finding, expansion has to wait. The existing response process remains in use while those gaps are repaired. A better observer or clearer runbook may answer the next question more effectively than changing the model.

Can another engineer explain the verdict?

Return to the checkout exercise for the first review. Give an engineer who did not build the agent its starting evidence, the evaluator’s criteria, configuration versions, action record, independent service observations, all trial results and the proposed stop procedure. They should be able to explain the verdict without relying on the agent’s declaration of completion.

In the checkout exercise, that record can preserve both useful diagnosis and unproved recovery. The next engineer knows which configuration change deserves attention and why the failing checkout request keeps the incident open. More demanding cases can then build on an evaluation that distinguishes progress in the investigation from restoration of the service.