Shadow-test the routing decision before replacing the log classifier

Ninety routine log events and ten deployment regressions arrive. A classifier calls all hundred routine. Its 90% accuracy is real; so is its failure to send any regression to an investigator.

This illustrative set explains why a routing migration needs more than an overall score. In the hypothetical service here, logs go to expected noise, dependency failure, deployment regression or security review. A new language-model classifier may use more context, but it can also time out or abstain, leaving a human decision to make.

Keep the event before either label is applied

Take one invented record: a checkout request timed out in the west region just after deployment r42. One parser keeps the message but drops r42. Another keeps the deployment revision. The classifiers now see different evidence, so different labels cannot tell you which classifier reasoned better.

Save that original record before either path transforms it. Keep the message, service and region, event time, severity and deployment revision; preserve the request's trace information when available so a reviewer can follow it. The OpenTelemetry model gives these details homes in the body, resource and attributes, with separate timestamps, severity, trace context and instrumentation scope. A parser change should not masquerade as better classification.

In our record, r42 is a clue to investigate, not proof of cause. Removing it can nevertheless hide the question the reviewer needs to ask. The logging guide explains why such context belongs with the source event, not only with the route chosen afterward.

Use only approved sensitive data. Replace or hash identifiers consistently so records can still join to reviews; that alone does not make arbitrary sensitive data appropriate. Record schema and classifier versions. If an attribute changes meaning, inspect the new schema as its own slice.

Match the evaluation population to future work. An error-and-fatal router should not earn accuracy from informational events it never handles. A whole-stream router does need ordinary events to measure false escalation volume.

Make the reviewed label explainable

A dismissed dependency alert may have been harmless or already covered by another incident. Keep the reason and evidence, not only “noise.” Preserve reviewer disagreements rather than manufacturing certainty.

Return to the ninety routine events and ten regressions. A random handful could contain only routine events and make the broken classifier look flawless. Deliberately include reviewed regression examples, then do the same for the other important destinations. Choosing examples within these groups is stratified sampling. Check different services, severities, record formats and kinds of failure so one familiar shape does not stand in for all of them.

Keep several records from one incident together when dividing development and evaluation data. Otherwise the prompt may be tuned using one near-copy and tested using its neighbor, a little like rehearsing the answer before the exam. Look for repeated templates even across different dates. Reserve a later group the prompt writer does not use; that held-out group asks whether the method works on work it was not adjusted to fit.

An illustrative release set might contain 2,000 events, at least 200 reviewed examples for each consequential queue and separate ambiguous cases. Those teaching counts are not a universal minimum. The useful property is that the reviewer can find each important kind of mistake and the current record formats, instead of trusting an average dominated by routine messages.

Let only the existing router control delivery

Send the invented west-region timeout to both classifiers, but let only the existing router decide where it goes in production. The candidate gets to propose a destination, not move the event. Beside the original record, keep its label, latency, timeout, parse result, model identity and any abstention reason. If the answer is missing or malformed, that belongs beside this same event too.

Now suppose the candidate's dependency stops answering. The production timeout still needs its usual route; it should not wait behind the experiment. A bounded shadow queue limits how much work can accumulate, a deadline shorter than record retention limits the wait, and shedding drops shadow work when the dependency fails. Count the eligible event before trying to enqueue it. Otherwise the event dropped at this first step vanishes from the comparison and makes the candidate look more available than it was.

The checkout event may produce a label, an abstention, a timeout, a parse failure or a dropped-before-evaluation result. Give it one primary shadow outcome and keep retries separately. Three attempts at this one timeout are still one event needing a decision. If the volume requires sampling, retain the selection rule and compare the selected population rather than quietly treating it as the whole stream.

An abstention leaves the candidate honest about what it cannot decide, but the event has not gone away. Someone's review queue receives that uncertainty. Watch that queue's size and age: coverage that requires reviewing half the stream may be supportable for one team and impossible for another.

Read the mistakes by their destination

In a separate illustrative group of ten deployment regressions, eight reach the regression queue and two disappear into noise. Four benign events arrive in that queue too. The investigator has twelve items to review, eight of them genuine regressions, while two regressions never reach them.

Those counts give the two useful fractions. The router found eight of the ten regressions: recall is 8/10, or 80%. Eight of the twelve routed items belong there: precision is 8/12, about 67%. The two missed regressions and four needless reviews are what the people using the queue experience.

Lay out the reviewed class against the chosen class for every event and you have a confusion matrix. The scikit-learn metrics include this view with precision, recall, F-score and support. A macro average gives each class equal weight; a weighted average follows how often each class appears. Both can summarize the table, but neither decides what a missed security event costs the team.

Set class-specific limits for missed events, added review, latency and fallback. Security review may accept extra work for stronger coverage. Automated suppression needs stronger evidence because a mistaken suppression hides the observation.

Promote the workflow that the paired records support

Compare the complete route from eligible event to usable decision, including queues, inference, parsing, abstention, retries and human work. Keep current and candidate outcomes on the same IDs so changing traffic cannot masquerade as better classification.

After shadow review, a canary can start with one reversible routing queue and capped authority, retaining the current rules as fallback. Suppression, deletion or automatic escalation needs explicit rollback and reconciliation for earlier decisions.

The incident-response evaluation guide provides related boundaries. A migration is supported when required classes, latency, review capacity and fallback improve the actual decision, not merely when its score is higher.

Keep the ledger after promotion and inspect class, abstention, deployment, schema and model changes. Drift should reopen the decision. The initial result might sensibly improve dependency/noise routing for two services while keeping regression and security paths unchanged. That narrower benefit is useful when the records show both who now receives the failure and who still carries the review.