Shadow-testing an AI log classifier before migration

A log classifier should not take over production routing because its aggregate accuracy looks good. Run the candidate beside the current path, keep its decisions non-authoritative, and compare both systems against reviewed outcomes from the traffic that operators actually see. The release question is not simply whether the new model predicts more labels correctly. It is whether it improves a specific operational decision without hiding failures in rare, expensive classes.

Consider a hypothetical platform team that routes log events into four queues: expected application noise, dependency failure, deployment regression, and security review. The current rules are imperfect, especially when a dependency timeout and a deployment error appear together. A new language-model classifier may read more context, but it also introduces variable latency, model-version changes, and an abstention decision the old rules never needed.

ChatGPT is most useful in an SRE workflow when the output has a factual boundary the responder can inspect. For this migration, that boundary is the original event, the candidate label, the current label, and the reviewed disposition. A score without those records cannot tell the team which mistakes it is accepting.

Preserve the event before comparing labels

Start with a stable input record. The OpenTelemetry logs data model separates the event body from resource, attributes, severity, trace identifiers, timestamps, and the instrumentation scope. Preserve those fields before either classifier transforms the event. Otherwise a parser change can look like a model improvement, or the candidate can appear to understand context that the baseline never received.

Keep sensitive fields out of the evaluation set unless their use has been explicitly approved. Hash or replace account identifiers consistently so the same event can still be joined to its review record. Record the input schema version and classifier version with every prediction. If a deployment changes the meaning of an attribute, treat the new schema as a separate slice rather than mixing it into the earlier result.

The comparison population should match the intended decision. If the classifier will route only error and fatal events, do not inflate its score with thousands of informational records it will never see. If it will process all logs, retain the ordinary traffic because that is where false escalation volume comes from.

Build a reviewed set from operational consequences

The site's logging guide explains why useful events need enough context to support investigation, not merely more volume.

A reviewed label is not automatically ground truth. An operator may close an alert because another system already identified the incident, or because the event was harmless in that environment. Capture the reason for the disposition and the evidence available at the time. When reviewers disagree, retain the disagreement instead of forcing certainty into the dataset.

Sample by consequence as well as frequency. Random sampling usually produces many common benign events and too few deployment regressions or security-relevant records. Add stratified samples for each class, service, severity range, schema version, and known failure family. Keep a time-based holdout so repeated templates from one incident do not appear in both the development and release sets.

For the hypothetical router, the release set might contain 2,000 events, including at least 200 reviewed examples for each consequential queue and a separate set of ambiguous events. That number is illustrative, not a universal minimum. The useful requirement is enough support to show the mistakes in each decision class and enough recent traffic to expose schema drift.

Run the candidate in shadow mode

Shadow mode sends the same eligible event to the current path and the candidate, but only the current path can change routing. Persist the candidate response, latency, timeout, parse result, model identifier, and abstention reason. A missing or malformed answer is an outcome, not a row to discard.

Do not let the shadow call delay production delivery. Put it behind a bounded queue, enforce a deadline shorter than the record-retention window, and shed shadow work when its dependency is unhealthy. Count every eligible event before enqueueing so dropped evaluations remain visible. If sampling is necessary, record the sampling rule and compare only populations selected by that rule.

The candidate should be able to abstain when evidence is insufficient. An abstention can be safer than a confident wrong route, but it creates review work. Measure that queue and its age. A model that improves per-class recall by sending half the stream to human review has changed the staffing problem rather than solved it.

Read the confusion matrix by consequence

Scikit-learn's classification metrics expose precision, recall, F-score, support, and a confusion matrix for each class. Use them to locate error types, then translate those errors into operational effects. Macro averages give each class equal weight; weighted averages follow class frequency. Neither decides how expensive one missed security event is compared with twenty extra dependency reviews.

For the deployment-regression class, recall asks how many reviewed regressions the candidate found. Precision asks how many events routed to that queue were actually reviewed as regressions. The false-negative row tells you where missed regressions went. If they become expected noise, the model can silence the exact evidence the team needs during a rollout.

Use a decision table beside the statistical report. For each class, record the allowed false-negative rate, acceptable extra review volume, latency deadline, and fallback. These are policy choices based on the workflow, not properties of the model. A security-review route may favor recall and tolerate more review. An automated suppression path should demand much stronger evidence because a false positive there hides data.

Compare completed decisions, not only model calls

Measure the complete path from eligible event to usable routing decision. Include queue delay, inference time, parse failures, abstentions, retries, and reviewer effort. Keep the same offered event stream when comparing model versions. A faster test that silently drops large events or times out fewer calls because it received less traffic is not an improvement.

For each service and class, report eligible events, completed classifications, fallback decisions, abstentions, timeouts, and reviewed corrections. Then compare the current and candidate routes on the same event identifiers. This paired view separates workload changes from classifier changes and makes a disagreement easy to inspect.

A short canary can follow shadow evaluation, but its authority should remain narrow. Start with one reversible queue, maintain the current rules as fallback, and cap the number of records the candidate can redirect. If the candidate starts suppressing, deleting, or escalating records automatically, the release needs an explicit rollback and a way to reconcile decisions made before rollback.

Promote only when the workflow improves

The incident-response agent evaluation guide provides a related pattern for separating test cases, action boundaries, and recovery checks.

The migration passes when the candidate improves the named operational decision across required slices, stays within latency and review limits, and preserves a reliable fallback. It does not pass because one aggregate score is higher or because a demonstration classified a few memorable messages correctly.

Keep the shadow ledger after promotion. Sample accepted decisions, monitor class and abstention distributions, and compare them with deployment, schema, and model changes. Drift should reopen the decision, not merely add a dashboard annotation.

For the hypothetical platform team, the first promotion might authorize the candidate to distinguish dependency failures from expected noise for two services, while deployment regressions and security review remain on the existing path. That is a smaller launch than replacing the router, but it ties authority to evidence the team can inspect. The classifier earns a broader role by improving complete routing outcomes, not by producing a more impressive label.