On this page5 sections
Three generated tests expect the same exception. An untidy older test covers the retry that once created duplicate work. In this hypothetical repository, sorting by age or authorship would make cleanup easy and leave us with the wrong files. Before removing anything, ask what mistake each file can still catch.
A test might protect the promise that retries stop after three attempts. A runbook might help someone find out whether a deployment actually happened after its request timed out. Several files can serve that one decision. A long, polished document might serve no current decision at all. That is a more useful starting point than the amount of text on disk.
Find the people and systems that still use it
List the tests and documents, their owners and references, then look for the last change that affected their meaning. A recent formatting commit can put a fresh date on stale instructions. An old regression test can still guard code used every day. Git history helps uncover what happened; it does not choose which history is worth keeping.
Follow the references before moving the files. A test name may be wired into a CI filter, dashboard, quarantine list or release procedure. A document might be linked from an alert, onboarding checklist, support reply or code comment. That quiet page with hardly any traffic could be the page someone opens during the next incident.
Put related material together. Tests for expired authentication, instructions for refreshing a token and an on-call note about stale sessions may all protect the same boundary. Read them as a group and contradictions become easier to spot. Reviewing three isolated files can miss that they give three different answers to one question.
Generation history can guide where to look first. A large batch with no named reader, or examples that change only the nouns, deserves attention. It is not a deletion rule. A human-written test can be empty reassurance; a generated one can catch a real regression. Both need to earn their place in the same way.
Make the test catch the wrong answer
A test can execute the line that decides whether to retry without checking the decision. Line coverage tells us the line ran. Branch coverage can show which paths were taken. Neither establishes that an assertion would notice a wrong answer. Coverage.py's branch coverage explains the branch distinction. For cleanup, we still need to identify the behavior being protected and deliberately put it at risk in a safe test.
Try finishing “This test fails if…” with something a person relies on. “The mock returns a different value” may describe the machinery, not the promise. “A timed-out deployment is reported as failed before its destination is checked” describes a mistake that could lead a responder to deploy it twice.
Now challenge that promise. Temporarily change the implementation to a plausible wrong answer, or use mutation testing, which makes small code changes to see whether tests catch them. If a changed version still passes, inspect why. It may behave equivalently or sit outside the intended contract. A surviving change is a question to investigate, not an automatic verdict that the test is worthless.
Consider a hypothetical client with three generated retry tests. Each mocks a timeout, calls the same function and checks for an exception. The actual contract makes three distinctions: reads safe to repeat, called idempotent reads, may retry; writes whose outcome is unknown must stop for reconciliation; authentication failures must not retry. Replace the repetition with a small table of those decisions and distinct assertions. Keep the old tests until the replacement fails against each deliberately wrong policy. The new file should catch the errors the old three only appeared to cover.
Let someone follow the page to a result
Documents need a comparable trial. A link checker finds missing destinations, and MkDocs strict mode can turn warnings into build failures. That helps establish that a page builds and its links resolve. It does not tell us whether the person arriving there can make the decision the page promises to help with.
Picture that reader at the point of arrival. What brought them here? A runbook should connect its entry condition to observations, permitted actions and recognizable recovery. A design note should make clear which changed assumption would reopen the decision. A tutorial should leave the reader with something they can observe, not merely a reassuring final paragraph.
Walk through from a clean starting state. Follow every internal link, try the commands in the supported environment and compare documented defaults with current configuration. Use a safe scenario or fixture for incident material. Validating a paragraph is not a reason to try its destructive step against production.
Then follow the old entry points. An on-call alert that still links to a retired page has sent its reader to a dead end, however good the replacement may be. Choose one canonical page when two cover the same job, and preserve the route with a redirect or clear replacement note. The reader needs to reach the answer; they do not care which duplicate file won the cleanup.
Write down why this file stays or goes
Keep a file when its current claim is still useful and its check still challenges the code. Rewrite it when the behavior matters but the test or explanation is weak. Retire it when the promise is obsolete or a checked replacement now protects it. Each decision tells the next maintainer something more useful than “removed AI-generated content.”
Record the original path, protected claim, consumer search, challenge performed, replacement if any, reviewer, date and rollback note. In a large repository, merge small groups that protect one decision boundary at a time. If a retirement turns out to be wrong, that grouping makes it easier to find and reverse without unpicking the whole cleanup.
For pages that guide operational action, the site’s runbook template can help structure the replacement. The incident-response agent evaluation applies a related test to AI systems: give an independent observation a chance to contradict a plausible-looking answer.
If you cannot yet challenge a file’s protected claim, stop short of retiring it. Keep it, or quarantine it while the missing information is recovered. “Nobody remembers why this exists” is a good reason to ask around and inspect history. It is not evidence that the next person will never need it.
Check that the useful work survived the tidying
Ask another person to find the authoritative procedure and the tests guarding its important choices. Working links, fewer contradictions and checks that fail on meaningful mistakes should make that easier. A smaller file count is a bonus only if those jobs survived.
Back at the retry client, the useful replacement distinguishes all three wrong policies: mishandling the permitted read retry, repeating an unresolved write and retrying an authentication failure. Once it rejects them, the repeated tests can go. The repository is smaller, but more importantly, another engineer can explain what remained protected and why the removed files were no longer needed.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- Git loggit-scm.com
- Coverage.py's branch coveragecoverage.readthedocs.io
- MkDocs strict modewww.mkdocs.org
