Separate monitoring from alerting
The most common mistake is treating every monitored event as an alert. Monitoring is cheap and should be broad — watch the sources, the entities, the signal health, all of it. Alerting is expensive, because it spends human attention, and attention is the scarcest resource the team has. Those are different economics, and conflating them is what turns a useful system into a screen people have learned to ignore.
So I build them as distinct layers. Monitoring records and evaluates continuously; alerting fires against an explicit interruption budget, only when an event clears a bar that justifies pulling someone off what they're doing. The question for every alert is not 'did something happen?' but 'is this worth a human right now?' — and most monitored events, honestly, are not.
Score severity before you interrupt anyone
Between monitoring and alerting sits a scoring layer that decides what's worth surfacing. It de-duplicates — the same event arriving from five sources is one thing, not five — attributes and correlates related signals, and ranks by severity against the team's actual priorities. Only the events that survive that filter become interruptions; everything else stays observable but quiet.
This is what keeps precision high. Without a scoring layer, more sources just means more noise: analysts triage a firehose, confidence is implicit, and the same incident shows up repeatedly wearing different labels. With it, an alert is a credible, de-duplicated, ranked signal — something worth opening — and the team relearns to trust that when the system raises its hand, it's usually right.
Watch signal health, not just uptime
Standard monitoring checks whether the server is up. Investigation monitoring has to check whether the signal is healthy, which is a much subtler thing. If a source that normally returns two hundred records an hour suddenly returns twenty, the server is 'up' and the coverage is silently broken. That kind of drift is invisible to uptime checks and catastrophic to an investigation that assumes the source is being watched.
So I instrument the shape of the data itself: throughput per source, freshness, the rate of new entities, distribution shifts that suggest the target changed its structure. Those become monitored signals in their own right, with their own thresholds. The system should be able to tell the team 'we're still collecting, but this source degraded an hour ago' — because in intelligence work a silent coverage gap is often worse than an outage you can see.
Route and close the loop
An alert that fires into a void is wasted. Each one needs a destination — the right person or queue — and enough context attached that the recipient can act without reconstructing what happened. That means the alert carries its evidence and its correlation, so opening it means opening a case, not starting an investigation into the alert itself.
And the loop has to close. Alerts should be acknowledgeable, resolvable, and — importantly — tunable based on what actually turned out to matter. If a class of alert is consistently noise, that feedback should raise its severity bar automatically over time. A monitoring system that never learns from its own false positives will drift back into fatigue no matter how good the initial thresholds were.
Alert on absence, not just events
Some of the most important alerts are about things that stopped happening. A source that goes quiet, a feed that hasn't updated in a window it normally would, an entity that suddenly disappears from coverage — these are non-events, and event-driven alerting misses them completely because there's nothing to fire on. In an investigation, silence is frequently the signal: the target changed behavior, or the collection broke, and both matter.
So I build expectations into the monitoring — this source should produce data at roughly this rate, this feed should refresh within this window — and alert when reality falls short of the expectation. That turns 'nothing happened' from an invisible gap into a first-class, actionable signal. A system that can only tell you when something appears is blind to exactly the coverage failures that quietly undermine an investigation's conclusions.
None of this works without deciding, in advance, what happens when an alert fires. An alert with no owner, no runbook, and no escalation path is theater — it makes the system look responsive while nothing actually happens on the other end. So I treat the response side as part of the design: each class of alert has a destination, an expected action, and a path to escalate if it isn't handled, so that firing an alert reliably turns into someone doing something.
This is also what keeps the interruption budget honest over time. When every alert has an owner and a resolution, you can measure which classes actually led to action and which were noise, and feed that back into the severity scoring. The alerts that consistently matter get sharper; the ones that consistently don't get quieter or get redesigned. A monitoring system that closes that loop keeps earning the team's attention, while one that fires into an undefined process trains them, correctly, to ignore it — which is the exact failure the whole separation of monitoring and alerting was meant to prevent.