Auditing AI-generated tests and documentation before cleanup

Three generated tests all expect the same exception. An untidy older test exercises the one retry that once created duplicate work. In a hypothetical cleanup, deleting by age or authorship would keep the repetition and lose the memory. The useful question is what behavior would disappear with each file.

Start with the decision somebody needs to make. A test may promise that retries stop after three attempts. A runbook may help an operator discover whether a timed-out deployment actually ran. Several files can protect one decision, while a long document may no longer support any current work.

Inventory consumers, not just files

Build an inventory of tests, documents, owners and references, then look for the last change that mattered. A formatting commit can make an old page look fresh. An old regression test can still guard today’s code. Git history helps you find the story; it cannot decide the file’s value on your behalf.

Search for consumers. Test names may appear in CI filters, dashboards, quarantine lists, or release procedures. Documentation URLs may be linked from alerts, onboarding checklists, support replies, or code comments. A page with little web traffic can still sit on a critical incident path. Record those relationships before moving anything.

Group the inventory by protected decision. Authentication expiry tests, token-refresh documentation, and an on-call note about stale sessions may all describe one operating boundary. Reviewing them together exposes contradiction and duplication more reliably than a file-by-file pass.

How a file was generated can help choose what to inspect first. A large batch with no named consumer or examples that differ only in nouns deserves attention. It does not decide what to delete. The test written by a person must still protect behavior, and a generated test may do so.

Translate coverage into protected behavior

A test can run the line that decides whether to retry and never check its answer. Line coverage records that execution; branch coverage can show whether different paths were taken. Neither tells you that the assertion would notice the wrong retry decision. That is why the next step is to name the behavior the test protects, then challenge that behavior deliberately. Related reading: Coverage.py's branch coverage.

Finish the sentence “This test fails if…” with a behavior somebody relies on. A changed mock value is rarely the whole promise. Reporting a timed-out deployment as failed before reading the destination is a decision a responder could act on incorrectly.

Challenge the sentence. Temporarily change the protected behavior, use a mutation-testing tool, or replace the implementation with a plausible wrong answer. A surviving mutation does not automatically make the test useless; the mutant may be equivalent or outside the intended contract. It does reveal that the current evidence did not distinguish the change. Review that result before deciding.

Consider a hypothetical client with three generated retry tests. All three mock a timeout, call the same function, and assert that an exception is raised. The production contract is more specific: idempotent reads may retry, writes with an unknown outcome must stop and request reconciliation, and authentication failures must not retry. The three files produce activity but protect only one vague claim. Rewrite them as a small decision table with distinct assertions. Retire the originals only after the replacement fails against each deliberately wrong policy.

Make documentation prove a reader action

Documentation needs a similar challenge. A link checker can find missing targets, and MkDocs strict mode can turn warnings into build failures. Those checks establish that the page can be built and followed. They do not prove that the reader can make the promised decision.

Picture the page’s next reader and the question that brought them there. A runbook should lead from its entry condition through evidence and permitted actions to a recognizable recovery. A design note should explain which assumption would reopen its decision. A tutorial should leave a result someone can observe.

Then walk the page from a clean state. Resolve every internal link, run commands in the supported environment, and compare stated defaults with current configuration. For incident material, use a safe scenario or fixture. Do not test destructive steps against production merely to validate prose.

Imagine an on-call alert still links to the page you just retired. The replacement may be excellent, but the responder has reached a dead end at exactly the moment they need it. When two pages cover the same job, choose the canonical one and preserve the old entry point with a redirect or a clear replacement note. The cleanup is complete when the reader’s route still works, not when the duplicate file disappears.

Choose keep, rewrite, or retire

Keeping a file preserves a current claim whose evidence still challenges the code. Rewriting preserves the needed behavior while repairing its weak test or explanation. Retirement removes an obsolete promise or one now protected by a checked replacement. Each outcome says more than “AI file removed.”

Record the decision beside the evidence. Include the original path, protected claim, consumer search, challenge performed, replacement if any, reviewer, date, and rollback note. For large repositories, merge cleanup in small groups by decision boundary. That makes a regression easier to trace and a mistaken retirement easier to reverse.

The site's runbook template is useful for documentation that governs operational action. The incident-response agent evaluation applies the same principle to AI systems: a plausible output is not acceptance evidence unless an independent observation can disagree with it.

Stop the cleanup when a protected claim cannot yet be challenged. That boundary changes the recommendation from retire to keep or quarantine until the missing evidence exists. “No one remembers why this file exists” is a reason to investigate, not proof that the file is safe to delete.

Leave the repository more explainable

Afterward, ask someone to find the authoritative procedure and the tests guarding its important choices. Working links, fewer contradictions and tests that fail for meaningful mistakes help them answer. A smaller file count is welcome only when that work survived.

For the retry example, the tidy outcome is a decision table that catches all three mistakes: retrying the wrong read, repeating an unresolved write and retrying an authentication failure. Keep the original tests until the replacement rejects those deliberately wrong policies. Then removal has a reason another engineer can inspect: the behavior survived, while the duplication did not.