Designing monitoring that earns its alerts

Monitoring fails in a predictable direction. It starts useful. Checks accumulate, because adding one is easy and nobody is accountable for the total. Within a year the team has learned to dismiss it: the dashboard is green-ish, the alert channel scrolls past unread, and when something genuinely breaks the signal is indistinguishable from the noise that has been arriving all month.

The failure is not technical. Every individual check was correct. The failure is that they were all allowed to alert at the same volume, and volume is the only thing a human actually responds to.

Here is how we design it, and why each decision is made the way it is.

Three tiers, and only one of them interrupts anybody

Recorded. The largest category by far. Disk at 60%. A service restarted. A certificate renewed. This is data you want when investigating something, and it should never generate a message. If it is genuinely useful it will be on a page somebody chooses to open, and if nobody ever opens that page, that is useful information too.

Raises a ticket. Something is wrong and a person must deal with it, but not this minute. Pending security updates. Disk crossing 85%. A backup that has not run in a week. A certificate expiring inside a fortnight. This enters the normal queue with an owner, a history, and the same handling as any other request — which means it cannot be silently forgotten, and it also cannot wake anyone.

Wakes someone. Reserved for conditions where waiting makes the outcome materially worse: a monitored system unreachable, a service that should be running and is not, a filesystem about to fill. In our platform this list is deliberately short, and we defend its shortness. Every proposed addition has to answer one question — is an engineer losing sleep proportionate to this? Most of the time the honest answer is no.

We built the split into the product rather than leaving it to convention. Critical issues raise a ticket automatically; warnings do not. That is not a configuration option, because the moment it becomes one somebody sets warnings to page during an incident and never sets them back, and the whole system degrades to noise again over the following month.

The flapping problem, and why we solved it in the schema

The second failure mode is oscillation. A disk sitting at the threshold crosses it every few minutes. A service that restarts in a loop is down, then up, then down. Naive monitoring generates an alert on every transition, and by morning there are forty tickets describing one problem.

The usual fix is hysteresis in the application: remember the last state, suppress duplicates, add a cooldown. That works until somebody adds a second code path that raises issues, or a deployment resets the in-memory state, or two workers race.

We solved it in the database instead. There is a unique index on the combination of server, issue kind, and an "is open" flag — and because SQL permits any number of NULLs in a unique index, resolved issues set that flag to NULL and pile up freely, while only one row per kind can be open at a time. A second attempt to raise the same open issue cannot create a row; it updates the wording of the existing one and moves on.

The reason to put it there rather than in code is that the database is the one place every path must go through. It does not matter how many code paths raise issues, or how many workers run concurrently, or what a future developer forgets. The constraint holds because it is not optional.

A failing check must never report success

There is a quieter failure than noise, and it is worse. Monitoring that cannot reach the thing it monitors, and reports green.

We have seen status pages showing all-clear because the collector had died an hour earlier. Everything was fine, as far as anything knew, because nothing knew anything. The page was not lying deliberately; it simply had no way to distinguish "no problems reported" from "no reports".

Our public status page says "Status unavailable" when it cannot reach the monitoring, and it says so in plain words: this page is not telling you everything is fine, it is telling you we cannot currently tell you. That is less reassuring and considerably more useful, and usefulness is the only property that makes a status page worth reading.

The same principle applies internally. A server that has stopped checking in cannot report that it has stopped checking in — so absence of data is itself a condition we monitor for, evaluated centrally rather than by the thing that has gone away.

Thresholds should be boring and shared

There is a strong temptation to make thresholds configurable per client, per server, per service. It sounds like flexibility. In practice it becomes the place where an alert quietly gets turned off during a noisy week, and nobody remembers why the threshold on that one machine is 98%.

We use the same numbers everywhere. Disk warns at 85% and is critical at 95%. Memory warns at 95%. A certificate warns at fourteen days. These are not sophisticated, and sophistication is not what makes monitoring work — consistency is. An engineer who knows the numbers without looking them up can reason about any machine in the estate.

What "instrumented" actually commits us to

We tell clients that if we manage something, we monitor it. That is a real commitment with a real cost, and it is worth being precise about what it means.

It means the estate has an inventory, and the inventory is generated rather than maintained by hand, because a hand-maintained inventory is wrong within a month. It means a machine added to the estate is monitored by default rather than when somebody remembers. It means the monitoring is checked — we alert on the absence of check-ins precisely because monitoring that silently stops is worse than none, since it comes with a false sense of coverage.

And it means the alerts go somewhere a human being reads. An alert delivered to a channel nobody watches is not monitoring, it is record-keeping with extra steps.

The test

If your team has a filter rule for your monitoring alerts, the monitoring is already broken.

Not because the filter is wrong. The filter was the rational response to a system that had stopped distinguishing between important and merely true. The person who wrote it made a reasonable decision, and the system made it necessary. Fix the system.