On this page5 sections
A hundred log events arrive: ninety routine messages and ten deployment regressions. The new classifier calls every one routine. Its score is 90%. The investigator waiting for the regressions receives nothing.
That invented result is a useful starting point for changing a log router. In the service we will follow, events go to expected noise, dependency failure, deployment regression or security review. A language model may make better use of the surrounding details than the current classifier. It may also return no answer at all. Before letting it move anyone’s work, run it beside the existing router and compare where the same events would go.
Keep the revision number attached
Imagine a checkout request timing out in the west region just after deployment r42. One parser saves the message but drops r42; another keeps the revision. Even excellent classifiers could choose different destinations from those two records. We would be comparing what they were shown as much as what they made of it.
Save the original event before either path reshapes it. Keep its message, service, region, event time, severity and deployment revision, plus the request’s trace information when available. That lets a reviewer follow the request through related work. The OpenTelemetry model provides places for these details in the body, resource and attributes, alongside separate timestamps, severity, trace context and instrumentation scope. Both classifiers should begin with the same account.
The timing of r42 is a reason to investigate, not proof that the deployment broke checkout. Still, losing that detail removes a question someone ought to ask. As the logging guide explains, context should travel with the source event rather than live only inside the destination chosen for it.
Give the comparison only sensitive data approved for that use. If identifiers must be replaced or hashed, do it consistently so a reviewer can join related records. Hashing is not permission to use otherwise inappropriate data. Save the schema and classifier versions, too. An attribute whose meaning changes deserves its own comparison, rather than being mixed unnoticed with the old format.
Choose the events the router will actually handle. A system that routes only error and fatal messages should not collect easy points from informational messages outside its job. A router responsible for the whole stream does need those ordinary messages: they reveal how much needless escalation it would create.
Find out why the reviewer called it noise
A dependency alert was dismissed. Was it harmless, or had another incident already brought someone to the problem? Both can end with a closed item, but they teach the classifier different things. Retain the reason and supporting records. When reviewers disagree, keep that disagreement visible instead of turning it into a confident label nobody quite endorsed.
Our ninety routine events could fill a small random sample without a single regression. The broken classifier would then look perfect. Deliberately choose reviewed examples from the regression group and the other important destinations. That is stratified sampling: choose within meaningful groups so the smaller ones get seen. Include different services, severities, formats and failure types, not just variations of one familiar message.
Several logs from one incident should stay together when you split material for development and evaluation. Tuning the prompt on one record and testing on its near-copy gives the method an answer it has almost seen already. Repeated templates can do the same thing across different dates. Set aside a later group the prompt writer does not use, then test on that held-out group to see how the method handles work it was not adjusted to fit.
For illustration, a release set could hold 2,000 events, with at least 200 reviewed examples for each consequential queue and separate ambiguous cases. Those counts are teaching choices, not a universal minimum. What matters is being able to inspect each important mistake and current record format, including the ones a pile of routine messages would otherwise conceal.
Let the candidate suggest, not deliver
Send the west-region checkout timeout to both classifiers. Only the current router decides where the production event goes. The candidate writes down its suggestion beside the original record, along with latency, timeouts, parsing result, model identity and any reason for declining to choose a category. A missing or malformed answer belongs with that event as well.
Suppose the candidate’s dependency stops responding. The real checkout event must still take its usual route rather than wait for the experiment. Give shadow work a bounded queue, a deadline shorter than record retention, and a way to drop that work when the dependency fails. Count the eligible event before enqueueing it. Otherwise a full queue can discard an event before it reaches the comparison, making the candidate look available precisely when it was not.
Give each event one main shadow outcome: a label, an abstention, a timeout, a parse failure or a drop before evaluation. Keep retries separately. Three attempts to classify the checkout timeout are still one event waiting for a decision. If the volume requires sampling, retain the selection rule and report on that selected population, not as though every event entered the experiment.
An abstention means the candidate declines to choose. That can be the right response, but a person still has to handle the event. Watch the receiving review queue’s size and age. A system that asks humans about half the stream may fit a small team’s workload or bury it, depending on what arrives and who is available.
Count what reaches the investigator, and what does not
Take a separate example with ten deployment regressions. Eight reach the regression queue; two are called noise. Four benign events also land in the regression queue. The investigator now has twelve items, eight of them real regressions, while two actual problems have gone elsewhere.
We can describe those two experiences with two fractions. Finding eight of the ten regressions gives recall of 8/10, or 80%. Having eight genuine regressions among twelve routed items gives precision of 8/12, about 67%. The first fraction describes what was caught. The second describes how much of the queue belongs there. Two missed problems and four unnecessary reviews make the numbers tangible.
Put each reviewed class beside the class the router chose, and arrange the counts in a table. This is a confusion matrix: it shows which destinations get confused with which others. The scikit-learn metrics include it alongside precision, recall, F-score and support. When summarizing across classes, a macro average weights each class equally; a weighted average follows their frequencies. Neither average can decide how much a missed security event matters to this team.
Set limits for each class: missed events, added review, latency and fallback. The security queue may willingly take extra reviews to catch more genuine problems. Automatic suppression deserves a stricter case because the mistake removes an observation from view instead of merely sending it to the wrong person.
Give the new router a small, reversible job
Compare the whole journey from eligible event to usable decision. Queues, model calls, parsing, abstentions, retries and human review all belong in that account. Keep current and candidate outcomes on the same event IDs. Then a change in the incoming traffic cannot quietly take credit for a better classifier.
After shadow review, a small live trial, or canary, can give the candidate one reversible routing queue and limited authority. Retain the current rules as fallback. Suppression, deletion and automatic escalation need explicit rollback and a way to reconcile decisions already made; switching back does not undo yesterday’s actions.
The incident-response evaluation guide develops related boundaries. Here the decision depends on the required classes, latency, people available for review and fallback behavior. A higher score helps only when that combination gives the receiving team a better decision to work with.
Keep the comparison ledger after promotion. Changes in classes, abstentions, deployments, schemas or models should prompt another look. The first useful result may be quite small: better dependency/noise routing for two services, with regression and security routes left alone. Now the team can point to the events that reach the right person sooner, and still see the work that needs a human. That is a much more informative success than our original 90%.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- OpenTelemetry logs data modelopentelemetry.io
- Scikit-learn's classification metricsscikit-learn.org
