From an anomaly score to a useful service signal

A sharp traffic drop is unusual only in relation to an expectation. It might signal a failure, a scheduled shutdown or a customer changing how they use the service. An anomaly detector helps identify the departure; the surrounding workflow determines whether the finding merits investigation, a page or an intervention.

Choose that initial role before choosing a complex model. A review queue can accommodate uncertain findings and supply context. An overnight interruption or automatic mitigation needs a clearer response and stronger evidence. The evaluation should measure whether the detector earns the role being proposed for it.

What the detector expects to see

A fixed threshold compares a value with a defined boundary, such as a known capacity limit. A statistical or learned baseline can represent changing patterns over time or relationships among signals, such as strong weekly variation. The more complex option is useful only if it captures behavior the simpler comparison misses in a way that matters to the service.

The training or baseline data also need interpretation. They can contain incidents, measurement errors and exceptional load. Establish how the detector treats new services, missing values, seasonal changes and legitimate shifts after deployment. Those choices determine whether a later deviation means something unusual happened or the model's expectation has become inappropriate.

Eight detected incidents and twelve false investigations

Sample-level accuracy can conceal the failures that matter. A detector may classify most time points correctly while missing a small number of important incidents. Review detected and missed incidents, false investigations and delay from impact to a usable signal. Fifty flagged samples in the same failure do not become fifty separate successful detections.

Keep an evaluation period separate from the data used to tune thresholds. Otherwise information from the test period can leak into the decisions being assessed and make historical performance look better than it would have been at the time. Compare against a simple baseline too, so the added model and maintenance work have a benefit to justify them.

For an illustrative set with ten known incidents, suppose the detector identifies eight and produces twelve additional investigations unrelated to them. Event recall is 8 / 10, or 80%. If each flag corresponds to one reviewed event, precision is 8 / 20, or 40%. These figures describe coverage and the fraction of useful flags, but leave detection timing and responder effort unanswered.

Keep the event-matching rule and underlying counts with those percentages. They explain what counted as a detection and let the reviewer inspect the twelve false investigations. A late correct signal and an early actionable one can count equally in recall while offering very different operational value.

Count detected incidents and false investigations separately

Read diagram description

Illustrative event-level evaluation with one reviewed event per flag: eight of ten incidents are detected, while twelve additional flags are unrelated. Recall is 80%; precision is 40%. Timing and responder effort still need separate measures. Diagram labels: 10 known incidents: 8 detected + 2 missed; 20 reviewed flags: 8 true + 12 false; Recall: 8 / 10 = 80%: How many incidents were found?; Precision: 8 / 20 = 40%: How many flags matched incidents?.

A planned shutdown belongs in a different response path

Consider a hypothetical detector flagging a traffic drop during a planned customer shutdown. A review queue can establish that context without waking an engineer. The model may have correctly identified a departure from its usual baseline; the unsupported part would be treating that departure as evidence that someone must act immediately.

Check that the flag’s service and observation window match the planned shutdown before explaining the drop that way. The baseline describes the expected traffic; source freshness tells the reviewer how current the observation is. Those checks keep context from one planned change from being used to dismiss a different observation. The deviation and its cause remain different findings: a generated explanation still needs supporting evidence before it becomes the diagnosis.

Reviewed events reveal how long an investigation can wait and whether a responder has a useful action available. Those findings, together with the burden of false investigations, determine whether a signal belongs in a review queue or on a pager.

Rare severe failures can justify a noisy early warning even with sparse replay evidence. Document the consequence and the response the warning enables instead of maximizing precision at the expense of necessary coverage. The appropriate tradeoff depends on what would be missed, not just how many routine flags are inconvenient.

Review alert fatigue alongside missed coverage, then repeat evaluation when traffic, instrumentation or service behavior changes. A once-useful baseline can stop representing the conditions the team needs to distinguish. Maintaining that relationship is part of operating the detector.

For the planned shutdown, a review queue gives someone time to recover the missing context without an overnight page. Other events may leave a much shorter intervention window. The same detector can support both paths when its findings carry enough service and timing information for the response to fit the event.