What makes an anomaly worth investigating?

Imagine a service that goes quiet at midnight. Tonight the graph drops at eleven, while the people who normally use it still have an hour left. The count itself has been this low plenty of times. The hour is the part worth a second look.

An anomaly detector looks for a departure from what it expects. Give it a good expectation and it can bring that early drop to your attention. To decide whether to wake someone, you still need to know what customers might be losing and whether someone can help before morning.

The same traffic count means something different an hour later

A fixed threshold compares the reading with a number you chose. If a storage volume has a known limit, that may tell you enough. Give traffic the same treatment and a healthy night can cross the line simply because people went home. A time-aware baseline, an estimate of the traffic you usually see at that point in the week, can compare eleven tonight with other comparable nights. Now the early drop has a chance to stand out.

A baseline can also learn how several readings tend to move together, or how long a report usually takes at a given workload. That extra detail helps only while the comparison still fits the service. A new traffic schedule can make healthy behavior look odd for weeks. A gap in collection can make a missing observation look like a drop if missing data is turned into zero.

Even with fresh data and a good baseline, the size of the drop tells you how far the reading moved from expectation. To learn how much it matters, look at what people were trying to do. A quiet hour during a scheduled closure and a quiet hour caused by a broken login can leave much the same line on the graph.

Fifty flagged samples can describe one incident

To find out what a detector would contribute, start with a period you held back while choosing its baseline and threshold. When the samples are ordered by time, fit on earlier data and evaluate on later data. Otherwise information from the future can leak into the expectation you are trying to test. Compare with a simpler rule on the same period, so the extra work of maintaining the detector has something concrete to earn.

Count the incidents people needed to find and the investigations the detector would have opened. Decide in advance how close a flag must land to a known incident and how repeated flags will be grouped. Fifty flagged minutes from a single outage give the on-call engineer one incident to respond to.

Suppose ten known incidents occurred in an illustrative replay. Eight received a matching flag, two went unnoticed, and twelve additional flags pointed to unrelated activity. With one reviewed event per flag and at most one counted match per incident, recall is 8 out of 10, or 80%. Precision is 8 out of 20, or 40%. The first number tells you how many known incidents the detector found. The second tells you what share of the reviewed flags matched a known incident under that rule.

What the detector caught and what it sent for review

Read diagram description

Ten known incidents split into eight detected and two missed. Twenty reviewed flags split into eight matches and twelve false positives. Recall is 8 out of 10, or 80%. Precision is 8 out of 20, or 40%. The diagram does not measure detection time or investigation effort.

Those counts still leave out the clock. A flag that arrives after customers have recovered can count as a match in a retrospective review. Record how long it took to get a useful signal and how much work the false investigations took, so that late flag never looks equivalent to one that helped somebody intervene.

Who needs to see the early drop?

Back at eleven, someone finds a planned customer shutdown that might explain the quiet. A staffed review queue gives them a place to check the service, the hours and the freshness of the traffic data. A generated summary can point to the shutdown record, but the record still has to match the actual observation.

If customer work is failing and waiting would take away a useful response, page someone who can act in time. Review the cost of false investigations alongside that response. A rare failure with severe consequences may still need an early warning even when it appears too seldom in replays to estimate coverage well. Keep the warning tied to the action it enables, and take another look at how it reaches the responder when traffic, instrumentation or staffing changes.

Tonight the service and the scheduled hours match. The early quiet belongs in the review queue with that explanation attached. Next time the line may fall the same way while customers are still trying to sign in. The detector can bring both nights to your attention. The service evidence tells you which night someone needs to answer.