On this page8 sections
“Checkout has recovered.” The agent has found a suspicious configuration change and proposed a reasonable rollback. An engineer reading its answer might be ready to move on. Then someone tries to buy something. The request fails. In this hypothetical exercise, the agent has offered a useful diagnosis and a recovery claim that the service cannot support.
Keep both findings. The diagnosis deserves a look; the failed purchase deserves to stop the success claim. An incident-agent evaluation needs an observer outside the agent to check the service, with separate judgments for investigation, permitted actions and the result. Otherwise a convincing account of a repair can take the place of the repair itself.
Make the first job small enough to follow from beginning to end. The thing being tested is the working combination of model, retrieved information, tools, permissions, retries and completion checks. A conversation with the model alone leaves out the moment a tool times out or a requested write is refused. Those moments may decide what happens to checkout.
Decide what a useful first answer looks like
Ask for something more inspectable than “handle incidents.” For this trial, the job might be to identify the configuration change most likely to explain the failures, point to the supporting observations and propose the next permitted check. Another engineer can follow that trail without having to agree that the answer sounds confident.
The finish also depends on the permissions. A read-only assistant might do its job well by handing over a supported explanation and making its uncertainty clear. One allowed to change infrastructure owes the evaluator an account of what happened to the service afterward. If that account is missing, the assistant cannot skip ahead to “recovered.”
Anthropic's January 9, 2026 evaluation guidance separates the task, repeated attempts, interaction record and final outcome. Our rollback record and our attempted purchase belong in different places. Google's September 19 infrastructure-security account makes operational agents worth examining, but Google's deployment experience cannot tell us whether this checkout trial passed.
Give the agent the clues, not the ending
Begin with incidents whose facts can be reconstructed after sensitive details are removed. Five sound cases can reveal a defect worth fixing. They cannot establish broad reliability. Add the next case because it asks an important new question about the proposed job, not just because a larger collection looks more convincing.
Our checkout exercise needs two packets. Give the agent the logs, measurements, configuration history, dependency information and runbook excerpts available when the decision had to be made. Keep the expected facts, acceptable alternatives, forbidden actions and success checks in the evaluator's packet, outside the agent's readable workspace.
A finished postmortem goes in that second packet too. It knows how the story ended. Reconstruct the earlier view, then have an incident participant check that it still contains enough clues to solve the problem. Hiding the answer is fair; accidentally hiding every route to it is not.
| Change one part of the case | Question for the trial | What a useful response would show |
|---|---|---|
| A bad setting with a clear change record | Can it connect the symptom to a plausible cause? | The right resource and observations that support the link |
| The same symptom, but old measurements | Does it notice what the information cannot show? | Named uncertainty and a request for fresh data |
| A proposed repair beyond its allowed scope | Does the permission boundary hold? | No forbidden action, with a useful escalation |
| A write whose response never arrives | Can it handle an unresolved outcome? | Inspection of the destination before another write |
| The alert clears; checkout still fails | Does it check recovery independently? | Recovery still marked unconfirmed |
These variations should not all reward the same response. Asking for new measurements makes sense when the old ones are stale. Missing a clear configuration change is a different problem. Allow an explanation the facts support, even when it is not the sentence the evaluator expected to read.
Let someone else try checkout
Put this failed checkout in a disposable service or isolated test fixtures, with synthetic data and credentials that cannot reach production. Keep the intended tool descriptions and permission checks: we want to see those boundaries work. Restore the starting state after every trial. Otherwise a later agent may walk into a checkout that its predecessor repaired and get the credit.
When the agent requests a rollback, the execution boundary records both the request and its result. A separate observer tries checkout. Now the two records can disagree, as they do in our opening, without one erasing the other. The model's recovered: true belongs among the claims to grade. It must not give the agent access to edit the observation file that decides success.
Keep enough detail to find this particular failed purchase again: model, prompt, tool-schema and case revisions, environment image, timeouts, concurrency and resource limits. Anthropic's infrastructure-noise study shows why the resources around an agent matter to its results. If a constrained host caused the failure, identify it. Asking the team to fix the diagnosis would send them to the wrong part of the system.
And if the proposed rollback names the wrong, unpermitted resource, block and record the write before it runs. The execution-contract guide describes that boundary. A reviewer deducting points afterward cannot undo the change that the test allowed through.
Separate the check from the judgment
Some questions here have mechanical answers: did the agent attempt a forbidden action, and did the independent request meet the agreed condition? Others need an experienced person: was the investigation reasonable given the incomplete information? Before using a model to help with that judgment, compare its decisions with service experts. Keep a sensible diagnosis and a failed purchase as separate findings.
The downloadable Python grader and sample observation show the mechanical part using only Python's standard library. The grader reads evaluator-owned observations, keeps absent evidence unknown and refuses to let apparent recovery cancel out a forbidden-action attempt. It does not call a model or operate infrastructure.
python3 incident-agent-grader.py --self-test
python3 incident-agent-grader.py sample-observation.json
Here is a toy rollback record from an independent observer. It checks the user path, changes outside the intended work and whether the action record is complete:
{
"trial_id": "rollback-example-1",
"audit_complete": true,
"forbidden_action_attempted": false,
"user_path_ok": true,
"collateral_change": false
}
This particular toy record says the user-path check succeeded. It is not the failed purchase from the opening. For that purchase, the observer would record a false user-path result; if no observation arrived, it would remain unknown. A real service also needs an agreed observation window, traffic coverage and error limits. One request can show who owns the observation without certifying a production recovery.
The grader returns pass, fail or unknown, with reasons. Picture an illustrative batch of 20 trials: 12 pass, three fail and five remain unknown. Twelve of 20 have verified success, or 60%. Twelve of the 15 resolved trials passed, or 80%. Those are answers to different questions. Reporting the second does not resolve the five missing outcomes. The completeness lab follows that difference in more detail.
Count the attempts that did not help
Show the valid trials, independently verified successes, violations, unknowns, time and cost for each configuration and kind of case. The checkout agent may be good at a clear change record and poor at recognizing stale measurements. A single combined score can conceal that useful distinction.
Repeat cases under the same conditions and keep every attempt. Two passes in three tries tell us something the best reply alone cannot: the behavior varies. A few repeats can uncover that instability, but they do not estimate the risk of a rare failure precisely.
Suppose those 20 illustrative trials incur $12 in model and tool charges and produce 12 verified successes. That is $1 per verified success. Failed attempts and retries still belong in the $12. Engineering and review time are excluded unless measured and added, so this is not the total cost of operating the agent.
Now watch the responder receiving the answer. Can they follow its citation to the named resource, or do they first have to repair the explanation? Count that checking work too. The token-usage guide takes the accounting further. Here, the payoff is an engineer who can make the next decision with less work, not merely an inexpensive reply.
Use a benchmark for the question it asks
ITBench offers an open framework and Kubernetes incident scenarios. Check its prerequisites and isolate the deliberate faults used to create a scenario before adopting one. Running an executable benchmark can affect infrastructure. Reading its questions and running its setup are very different acts.
The ITBench-AA methodology tests root-cause analysis using saved Kubernetes incident snapshots and known answers. That can help select candidates for diagnosis. It does not, by itself, show what happens during a live repair, after a write loses its response, or when an agent must decide whether service has really returned.
A strong offline diagnosis result may earn the agent a try at our configuration fault. The local tools and independent observer still have to tell us whether its rollback reaches the right target and checkout recovers. Keep some cases out of prompt tuning, too, so improvements do not simply teach the familiar checkout story. New failures widen the questions; a stable subset lets the team check that an improvement has not broken earlier behavior.
Start with a suggestion, then consider a bounded action
The first useful role for this agent may be diagnosis alone. Start with offline replay and named owners for the evaluation, service and stop decision. If the results justify read-only shadow use alongside actual response, label its recommendations clearly as suggestions, not instructions to the on-call engineer. Finding the configuration clue does not grant permission to roll it back.
A later pilot with write access should allow one reversible operation, with explicit limits on its target and parameters. Canarying guidance offers the principle: expose a limited part of the service, observe it and use those observations to decide whether to continue. Passing the offline cases does not replace watching the live service.
Our unsupported recovery message can be a reason to stop that write-enabled pilot, even after a good diagnosis. So can an out-of-scope attempt, missing action records or an unresolved write. Stop new actions, reconcile those already in flight, then use established rollback or compensation where appropriate. Switching off the agent does not reverse a change it has already made.
Keep the checkout clues without keeping unnecessary private data. Use approved model and storage arrangements, remove secrets and customer details, restrict access to raw records and retain as little transcript material as needed. But do not replace every resource name with the same token: the evaluator still needs to distinguish the allowed rollback target from the forbidden one. The relationships are part of the case.
If the team cannot define success, reconstruct permitted actions or tell a missing observation from a negative result, postpone expansion. Keep the existing response process while those gaps are repaired. For this checkout exercise, improving the observer or runbook may do more than changing the model.
Make the verdict understandable to another engineer
Hand another engineer the starting packet, evaluator criteria, configuration versions, action record, independent observations, every trial result and the stop procedure. They should be able to explain the verdict without taking the agent's last message on trust.
That engineer can keep the useful configuration clue and reject “checkout has recovered.” The failed request is still there, along with a clear place to continue the investigation. The evaluation has done something valuable: it has kept a plausible answer from ending the work too soon.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- agent-evaluation guidancewww.anthropic.com
- September 18 account of agents in infrastructure securitycloud.google.com
- study of infrastructure noisewww.anthropic.com
- ITBenchgithub.com
- ITBench-AA methodologyartificialanalysis.ai
- Google’s canarying guidancesre.google
