Opens in a new tab
Commentary

The database recovered. The customer still cannot sign in.

In brief

When alerts share a headline, one successful repair can hide a second failure. Follow the remaining symptom through the group, the trial and the next responder.

5 min read

Is this the right starting point?
Before you read
Basic alerting, incident response and monitoring familiarity; deduplication and correlation are distinguished.
What you will take away
Evaluate alert grouping for retained failure coverage, separable mistaken groups, shadow-mode behavior and workable fallback.
Sources
Two lanterns stand beneath a translucent purple canopy, one dark and one lit, with a loose wick at left and a magnifying glass at right.
On this page4 sections

A responder repairs the database and watches the incident turn green. Then a customer tries to sign in. The login still fails.

In this hypothetical incident, the database ran out of capacity at about the same time that an unrelated software change broke authentication, the process that checks who is signing in. The alerting system put both problems under one headline. That looked helpful until fixing one problem appeared to close the other.

Grouping alerts can save people from investigating the same failure repeatedly. To keep that benefit, the group also needs a way to let a separate failure back out, carrying its history and a clear owner. The customer's next failed login is the detail that matters now.

Two ways to make the notification list shorter

Suppose the same database warning arrives again and again. A stable identifier can let the alerting system recognize those notifications as repeats of one condition. Removing that repetition is deduplication.

Now suppose several applications report connection errors while the database is failing. Those are different signals, but investigating them together may lead everyone to the shared dependency sooner. Grouping signals because their timing, dependencies or messages suggest a relationship is correlation.

Both approaches can leave a shorter list on the responder's screen. The second makes a larger claim: these different things may belong together. In our login example, their timing suggested a connection that the database repair did not bear out.

The responder should be able to open the group and find the login events, which users were affected, when the failures occurred and who is responsible for investigating them. If those details were thrown away, someone has to reconstruct the second incident before working on it. The tidy list has saved space by moving work into that person's afternoon.

Keep the replay running after the database recovers

A useful trial includes the satisfying case where one repair really does clear many alerts. It also needs independent incidents that happen together, several services slowed by a shared dependency, and failures reported by customers when the system's own measurements are sparse.

Replay each case using only what a responder could have known at that moment. Giving the grouping system the eventual postmortem explanation would make the exercise easier than the job.

For the database and login case, let the replay continue past the repair. Look for the surviving login failure and follow it to the person who would investigate. Can that person separate it from the recovered database problem without losing its records? This is where a trial that ended at the first successful repair would stop too soon.

Run the same cases through the previous routing as well. Compare the notifications removed, the people receiving the work, the symptoms still visible and the effort needed to split a mistaken group. Fewer notifications help when they mean less repeated investigation, rather than a longer search for the remaining problem.

That does not require reading every repeated warning one by one. Check distinct symptoms and affected groups of users. Representative checks can save time when you understand what they cover. If a sample cannot tell you whether another failure remains, widen the check instead of treating the unanswered question as a recovery.

Let the new grouping run beside the existing pages

Before letting the candidate hide notifications, keep the existing paging route running and let the candidate group copies of the same events. This is shadow mode: its decisions are available for comparison without removing the notifications responders already receive.

In our imagined incident, you could then examine why the candidate joined the login failure to the database problem while the original signals remain available. Disagreements become something the team can investigate during the trial, rather than something a customer has to discover after launch.

Decide what sends the system back to its fallback route: missing source events, out-of-date dependency information, failure of the grouping service itself, or a confirmed loss of failure coverage. A confidence score may help you interpret a grouping decision if you understand the score. It cannot tell you that an event arrived when the event never reached the system.

The fallback also needs people who can carry its notification volume and a working path for the events to reach them. Switching back to the old feed is less reassuring if the rotation is no longer prepared to receive it. Include that load in the trial.

Keep the history when the group changes

A service that once had one owner may later split into two. A grouping rule that used to send related work to the right person may now join work that needs different responders. Keep the rule and model versions, the records of suppressed events, and the reasons people split or overrode a group. Those details let the next responder understand how the headline was assembled.

Google's monitoring guidance emphasizes actionable symptoms. Here, the symptom that still needs action is the failed login. The on-call-load guide follows the other side of the decision: whether the shorter feed actually reduced the work people carry.

Resolving one member must not hide another failure
Resolving one member must not hide another failure. Illustrative over-grouping: database errors clear after mitigation while an independent authentication regression remains. Recheck group members before resolving the whole notification.
In this illustrative case, repairing the database clears one set of errors. The separate login failure still needs an investigation and an owner.
Follow the two failures through the repair

This conceptual drawing follows two failures placed under one headline. Database recovery clears only its own symptom; the authentication failure remains active. The drawing does not report a measured incident. Diagram labels: Database errors: Dependency incident; Authentication failures: Independent regression; One correlated group: A shared headline can imply one cause; Database recovers: This symptom clears; Authentication still fails: Keep investigation and ownership active.

At the end of this replay, the database can be marked recovered while the login investigation stays open. Someone has its records and knows it is theirs to pursue. That is a useful outcome for grouping: the first repair remains a success, and the customer still waiting to sign in has not disappeared behind it.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next