Checking AI incident-summary citations with Python

An incident summary recommends restarting a worker and cites the runbook. The quotation is real. It is also from release 2.3, while the affected workers run 2.4, whose procedure requires preserving a diagnostic buffer first. Following the citation has established where the sentence came from, but not whether it belongs in this decision.

This guide builds a small local check for that kind of mismatch. It compares a proposed citation with a reviewer-prepared source packet, checks the captured text and intended release, and leaves the interpretation for a person to inspect. The example is illustrative and synthetic, uses Python’s standard library, and makes no model or production calls. You can use it to learn what a citation check can establish before deciding whether to adapt it to an incident assistant.

The important source is not necessarily the one repeated most often. A current exception can outweigh several older success stories, so the packet must preserve the date, applicable version and passage that changes the decision. That judgment determines what we check here: a plausible link alone is too little, and a count of supporting documents would answer a different question.

Start with a source packet the model cannot redefine

First, choose the release you are investigating and capture the relevant source passages yourself. Keep the packet outside the assistant’s write access. The assistant can propose a source ID and quotation, but it must not supply the trusted copy against which its own answer is checked. Otherwise a fabricated passage could arrive on both sides of the comparison.

For the fictional worker, the packet contains a 2.3 instruction and a 2.4 exception. Both remain visible because the older instruction explains the disagreement. Their effective dates are recorded for review, not used as an automatic ranking rule: the newest document can still concern a different service or an unapproved proposal. The reviewer selects the applicable release explicitly.

Download the synthetic source packet, candidate citation and Python checker into one empty local directory. Use Python 3.11 or later. There are no third-party packages to install. Inspect the small files before running them, and start with these supplied fictional records rather than an actual incident transcript containing sensitive information.

Each source has a stable ID, exact captured text, declared version, effective date and SHA-256 digest. The digest makes a later change to that text detectable relative to the recorded value. It does not prove who wrote the document or whether the original capture was trustworthy. Store and review the packet accordingly. The Python hashlib reference documents the digest operation used by the example.

Run the citation that looks convincing

The candidate says, “The worker can be restarted immediately,” and quotes the fictional 2.3 procedure. The target release is a separate command-line argument, chosen by the reviewer rather than accepted from the candidate. From the directory containing the downloads, run:

python3 incident-citation-check.py --self-test
python3 incident-citation-check.py incident-citation-sources.json incident-citation-candidate.json 2.4

The self-test exercises the teaching logic, including expected failures. The second command should print citation-check-failed with the reason source version differs from reviewer target and exit with status 1. This is the useful disagreement: the quoted sentence exists, but its source does not match the release under review. Do not “repair” the test by changing the target to 2.3 unless the investigation really concerns that release.

The checker also rejects an unknown source ID, a changed captured-text digest and a quotation absent from the captured passage. It expects exactly three nonempty text fields in the candidate: claim, source_id and quote. Duplicate JSON keys are rejected rather than silently retaining the last value. The standard-library JSON documentation describes the decoding hook used for that check and recommends bounding untrusted input size; this teaching example accepts at most 64 KiB per file.

Those checks remove avoidable review work, but they do not choose a mitigation. They tell the reviewer why this particular citation cannot yet support the draft. Open the authoritative 2.4 procedure, confirm that its exception applies, and revise the summary so the diagnostic-preservation requirement remains beside the proposed restart.

A passing citation can still support the wrong claim

Change the candidate’s source to runbook-24 and its quotation to the exact 2.4 text in the packet. A suitable draft claim is “Check the diagnostic-preservation requirement before considering a restart.” Running the same command with target 2.4 should now return citation-integrity-passed. Its remaining fields explicitly require semantic review and say that source freshness and execution authorization have not been established.

Now try the more revealing test. Keep the correct 2.4 quotation, but change the claim back to “The worker can be restarted immediately.” The structural check still passes. The bytes and declared version match even though the claim discards the condition that matters. This deliberate counterexample is included in the self-test, so the limitation is visible in executable behavior rather than hidden in a disclaimer.

Reading the whole relevant passage is therefore essential. Ask whether the claim follows from it, whether an exception sits just outside the quoted span, and whether the source concerns the same component and failure condition. An exact-match checker cannot answer those questions. A model-based reviewer might help locate a disagreement, but its answer would also need evaluation rather than becoming an independent fact by virtue of using a second model.

The quotation minimum in this example is twenty characters, a teaching heuristic that prevents trivial matches such as a product name. It is not a meaningful universal evidence threshold. A longer quotation can still omit the next sentence, while a short configuration value can be decisive. Adapt extraction and review to the actual source format instead of treating this number as a safety rule.

Keep captured evidence separate from current state

The source packet is a fixed review input. Its recorded date does not tell the checker whether someone updated the runbook after capture, and the selected release does not establish what is currently deployed. Before consequential use, a reviewer still needs the maintained procedure, current service state and the existing action-approval process. A matching citation cannot replace any of them.

That distinction makes the example useful for offline regression tests. Keep the fictional conflict fixed, change a prompt or retrieval configuration, and inspect whether the new summary cites the appropriate source and preserves its qualification. Retain failures and unsupported claims as findings. Do not report the checker’s passing percentage as factual accuracy: it measures only the structural conditions the code actually examines.

If the goal is broader evaluation, the incident-response agent guide separates agent output from independently observed service results. For source disagreements, the NotebookLM comparison guide explains how to keep conflicting procedures visible. This checker supplies a smaller building block between those tasks: it makes a citation’s captured source and declared applicability inspectable.

In the worker example, the first useful result was a rejected 2.3 citation. The second was a passing 2.4 citation whose interpretation still needed review. Keeping those outcomes distinct lets an assistant shorten the incident record without quietly turning a historical instruction into today’s permission.