Seven Alerts, One Problem

Seven alarms before nine o'clock

You open your inbox and there are seven alerts waiting. Sessions are down. Conversions are down. Revenue is down. Average order value has moved. New users are down. Returning users are down. Events per session has dropped.

Seven subject lines, seven thresholds, seven separate emails. They all landed within the same minute, and they are all about one dataset.

This example is illustrative of a shape rather than a report of any particular morning, but the shape is the point and most people reading this have had that morning.

What it is not: seven problems

The natural response is to work the list. Open the first alert, pull the report, check whether the campaign changed, check whether the landing page changed. Then the second. Then the third.

That is a wasted morning, and it is a specific kind of wasted: you are looking for seven business explanations for something that has one technical cause. Each investigation comes back inconclusive, which sends you looking harder rather than looking elsewhere. By the time someone thinks to check whether the data arrived at all, half a day is gone.

The cost is not really the triage time. It is that the actual failure went unexamined while everyone examined its symptoms.

Simultaneity is the diagnosis

Here is the thing worth internalising, because it turns the noise into information.

Metrics do not coordinate with each other. Sources do. Sessions and average order value have no mechanism for moving together. They measure different things through different paths. Nothing connects them except the pipeline that feeds both.

So when they move together, at the same moment, on the same dataset, the correlation is not a coincidence to be explained. It is the finding. Something upstream of all of them changed, and the only thing upstream of all of them is the source.

One metric moving is a business event. Every metric moving together is an infrastructure event. That distinction is available to you the moment the alerts arrive, and per-metric alerting throws it away.

Two categories, two different first questions

The split matters operationally, not just conceptually, because it routes the work to different people.

A business event goes to whoever owns the campaign or the page. The first question is what changed in the market, the creative, the pricing or the audience. It is usually not urgent within the hour, and it rewards careful analysis.

An infrastructure event goes to whoever owns the pipeline. The first question is what changed in the feed: a deploy, a credential, a schema, a scheduled job. It is urgent within the hour, because every hour it continues is an hour of data you will have to caveat later.

Sending an infrastructure event to the campaign owner is not just inefficient. It hands the problem to the one person who has no way to fix it, and it does so with seven emails that each look like their job.

Why per-metric alerting cannot express this

Alerts are defined on measures. You pick a metric, set a condition, choose a recipient. That is the unit the tooling works in, and it is a sensible unit for the thing it was designed to do.

But a relationship between measures has nowhere to live in that model. There is no field for "this alert and that alert are about the same feed". So seven alerts arrive as seven independent facts, with seven subjects, and nothing on screen relating them. The correlation exists in the data and is destroyed at the moment of notification.

This is a design gap rather than negligence, and it is worth being fair about it. Plenty of monitoring platforms can group alerts; several do it well. The problem is that per-metric alerting as it is commonly configured does not, because grouping requires someone to have modelled the relationship in advance and almost nobody does that at setup time.

The rule: group by source

The rule is simple enough to implement in an afternoon. If more than half the metrics on a single dataset breach within the same window, do not send the individual alerts. Send one item that says the dataset is in trouble.

It needs a window rather than an exact timestamp, because checks do not all run in the same instant. A few minutes is usually enough. It needs a notion of which metrics belong to which dataset, which is the part most teams have to write down for the first time, and which is useful to have written down regardless.

What arrives is one alert with a different subject: not "conversions are down" but "nine of eleven metrics on this dataset breached at 08:14". That sentence routes itself. Nobody who reads it opens a campaign report.

Why this is the right cure for alert fatigue, and thresholds are the wrong one

Under a barrage of alerts the instinct is to raise the thresholds. Fewer emails, calmer inbox, problem apparently solved.

It is worth seeing exactly what that trade does. Raising a threshold reduces volume by reducing sensitivity. You get fewer alerts because you have agreed to be told about fewer things, including things you would have wanted to know. The system gets blinder while feeling calmer, and the feeling arrives immediately while the blindness shows up later.

Grouping reduces volume by adding structure. Seven emails become one, and no sensitivity is lost. Every breach is still detected; they are simply reported as what they are, which is one event.

Both approaches shrink the inbox. Only one of them leaves you able to see. This is not a criticism of anyone who has reached for the threshold dial under pressure, which is most people, because it is the control the interface puts in front of you.

The timestamp tell

The practical takeaway fits in a sentence. Look at the timestamps before you look at the metrics.

If the alerts are spread across the morning, they are probably about different things and deserve individual attention. If they match to the second, stop reading the metrics entirely and go look at what feeds them. The clock has already told you which kind of problem you have, and it is the only field in the alert that is not trying to describe a business.

This assumes the alerts are still running, which is a separate assumption worth checking, and one we covered in Nobody Monitors The Monitor. Seven alerts at once is a loud failure. The quiet one is the morning with no alerts at all, and no way to tell whether that means calm or dead.

It also assumes the numbers under the alerts were collected when they claim they were. If they were not, as in The Date Column Is Lying To You, the timestamps stop being reliable evidence about anything and the tell stops working.