Nobody Monitors The Monitor

You monitor the system. Nothing monitors the monitor.

Every alerting setup has the same blind spot, and it is not a gap anyone left by accident. You have checks watching your data, your tags, your jobs and your uptime. Almost nobody has anything watching the checks.

So when the alerting stops, it stops quietly. There is no alarm, because the thing that would raise the alarm is the thing that broke.

It fails silently by construction, not by neglect

This is worth stating plainly, because the instinct is to treat it as somebody's oversight. It is not. The mechanism that reports failures and the mechanism that failed are the same mechanism. When it goes down it takes its own reporting with it.

You cannot fix that by being more careful, or by adding another alert of the same kind. An alert defined on a system that is not running does not fire. It does not fire loudly, it does not fire late, it does not fire at all. There is no signal to miss.

It is a design property of alerting, and the fix has to come from outside the design rather than from more of it.

Silence has two causes

Here is the line the whole thing hangs on. A quiet dashboard has exactly two explanations, and threshold alerts cannot tell them apart.

Either it was a calm week and nothing breached, or the check died and nothing was evaluated. From the outside these produce identical output: no alerts, no incidents, a clean board. Whoever glances at it concludes things are fine, which is the correct reading of one of those two situations and a serious misreading of the other.

A quiet dashboard is not evidence of a quiet week. It is evidence of an absence of alerts, and an absence of alerts is not the same claim.

Three shapes this takes

These are patterns rather than incidents, and they are real in kind. Any team running scheduled checks will recognise at least one.

The scheduler stops. A job that was running nightly stops being scheduled. Nothing errors, because nothing runs. Often the first signal is a recovery notice: something comes back to life weeks later and announces that an outage has ended, and that is how everyone learns there was an outage.

A token expires. The credential behind a check lapses. If the client is written to treat an auth failure as an empty response, and many are, the check does not error. It returns zero results. Zero results looks like good news. The check reports that it found no problems, which is technically true and completely misleading.

A scan gets blocked. A crawler hits a bot wall, gets nothing back, and records nothing found. The absence of findings is written down as an absence of problems. This applies to our own scanning as much as anyone else's, and saying so is the only honest way to make the point.

Why threshold alerting cannot catch any of the three

An alert of the form "tell me if this drops below X" has to evaluate X against something. No data means no evaluation, and no evaluation means no alert. The absence of the input silences the alarm about the input.

This is the same structural gap as a zero that could mean two things, which we covered in Your Dashboard Cannot Tell You It Is Broken. There the reader could not tell a real zero from a missing measurement. Here the alerting cannot either, and it is making the decision on your behalf.

Lowering the threshold does not help. Raising it does not help. The threshold is not the part that is broken.

Heartbeats turn silence into a signal

The fix is unglamorous and it works. Instead of only emitting alarms, have each check emit a heartbeat: a short report saying it ran, what it looked at, and how many things it checked. Then alert on the absence of that heartbeat.

This inverts the failure mode. Previously silence meant nothing and could not be distinguished from health. Now silence is itself the alarm, because a working check produces a steady signal whether or not it finds anything wrong.

The pattern is sometimes called a dead man switch, which is worth one sentence of explanation rather than assuming it is familiar. It is a control that triggers when the regular signal stops, rather than when something unusual arrives. Same idea, applied to your monitoring.

A useful heartbeat says more than "alive". It says what was covered. A check that ran but examined nothing is a different problem from a check that did not run, and both are different from a check that ran, looked at everything, and found no issues.

Zero results and could-not-check must never render the same

This one sounds like a detail and is not. Two different states are routinely collapsed into one presentation, and every downstream consumer then inherits the confusion.

"We checked and found nothing" and "we could not check" mean opposite things. The first is a measurement. The second is a gap in coverage dressed up as a measurement. In a UI, in a report, and in an API response, they need distinct renderings, always.

Absence of findings is not a finding. An empty result is only meaningful if you can show the check actually ran and actually saw the thing it was checking. Otherwise a clean report is a claim about the checker's experience, not about the system.

The same applies to any figure carrying a date it did not earn, which we covered in The Date Column Is Lying To You. Confident presentation of a value nobody measured is a recurring failure, and it is always presentation covering for a missing distinction.

The thirty-second test

Pick your single most important alert. The one you would most want to fire. Then answer one question: if it had stopped working this morning, what would be different at five o'clock today?

If the honest answer is nothing, it is not monitoring. It is a thing that has historically produced emails.

You can run that test on every check you own in about half an hour, and the results are usually uncomfortable in a productive way. The checks that fail it are not necessarily the ones you would guess.

Monitoring is not the alerts

It is the assurance that the alerts are still running. Those are different products, and most teams have bought the first and assumed the second.

The good news is that the fix is small. A heartbeat per job, an alert on its absence, and a rendering rule that keeps zero and unknown apart. None of that requires a platform migration, and it converts your quietest failure mode into your loudest one.