Thirty-one alert rules and a notification channel that every member of the team had muted. That is a monitoring system that has been switched off by consensus, and no amount of adding rules to it improves anything.
The symptom
one month, by rule:
rule fired acted on
node_cpu > 80% for 5m 188 0
disk_used > 70% 94 0
http_5xx_rate > 0.1% for 1m 41 1
replica_lag > 5s 22 0
memory_used > 85% 18 0
queue_depth > 1000 14 0
... 22 more rules 31 1
checkout_error_rate > 2% 4 4
total 412 6One rule out of thirty-one has a hundred per cent action rate and it is the last one — the only one phrased in terms of something a customer would notice. Everything above it describes a machine, and a machine being busy is not by itself a problem.
Why it happens
Alerts are added after incidents, by the person who has just been surprised, and they are never removed after the underlying cause is fixed. Each one is individually reasonable and the set is unusable.
The fix
Counting first
# every firing is already recorded. the query exists.
sum by (alertname) (
count_over_time(ALERTS{alertstate="firing"}[30d])
)
# and the second number, which is not recorded anywhere:
# whether anybody did something. we added a reaction
# convention in the channel — ✅ for acted on, 🔇 for
# ignored — and counted by hand for a month.The action rate is the number that decides everything and no tool records it, so it has to be gathered deliberately. A month of emoji reactions is unscientific and produced a clear enough answer that nobody argued with the conclusion.
Deleting twenty-three rules
for each rule, one question: what would we have missed?
node_cpu > 80% nothing. every incident that
mattered also breached a latency
objective, which we now alert on.
disk_used > 70% nothing urgent — this is a
capacity report, not an alert.
→ moved to the daily summary,
with a separate 90% page.
replica_lag > 5s nothing. it is a symptom of a
long transaction, which shows up
as latency.
memory_used > 85% nothing. the JVM-shaped
assumption behind this rule does
not apply to php-fpm.
deleted 23. moved 3 to reports. rewrote 5.Writing down what would have been missed is what makes deletion defensible rather than reckless, and it is the artefact that survives the person who did it. Two of the twenty-three were genuinely hard to argue about and were kept for another quarter, then deleted.
Symptom over cause
# before: five rules describing possible causes
- alert: HighCpu
- alert: HighMemory
- alert: SlowQueries
- alert: QueueDepth
- alert: ReplicaLag
# after: one rule describing the promise
- alert: CheckoutLatencyObjectiveBurning
expr: |
histogram_quantile(0.95,
sum by (le) (rate(http_duration_seconds_bucket{
route="checkout"}[5m]))) > 1.5
for: 5m
annotations:
summary: "checkout p95 {{ $value | humanizeDuration }}"
runbook: "https://wiki.internal/runbooks/checkout-slow"The five cause-rules become dashboard panels, which is where they belong — they are what somebody looks at after the symptom alert fires. The symptom rule cannot tell you what is wrong and it can tell you that something is, which is the only thing an alert needs to do.
The runbook requirement
the policy: an alert may not be created without a
runbook link, and the runbook must contain at least one
imperative sentence.
what happened when it was applied to the survivors:
6 of 8 a runbook was written in under an hour
2 of 8 could not get past "investigate the cause"
→ both were deleted. "investigate" is a
dashboard, not a procedure.
and the effect on new alerts: 4 proposed in the
following quarter, 2 created. the other two died at
the runbook stage, correctly.Alerting on absence
# the category nobody adds: a thing that stopped
- alert: NightlyBackupDidNotRun
expr: |
time() - max(backup_last_success_timestamp_seconds) > 93600
for: 10m
annotations:
summary: "no successful backup in 26 hours"
- alert: MetricsStopped
expr: absent(up{job="app"}) or
(time() - max(app_heartbeat_timestamp) > 300)Every alert we had was for a value being too high, and none for a value not arriving. The second rule is the one that guards the first: a monitoring system reporting nothing looks identical to a healthy system, and that is the failure mode with no natural symptom.
Routing by severity, honestly
two destinations, and the rule for choosing:
page a human is woken. reserved for: the service
cannot serve, or money is being lost.
→ 3 rules.
channel somebody looks at it during working hours.
→ 5 rules.
and nothing else exists. an "informational" severity is
a dashboard panel with an alert's clothes on, and every
one we had was in the muted channel.Verifying it worked
# three months after
rules: 8 (was 31)
firings/month: 14 (was 412)
acted on: 12 (was 6)
action rate: 86% (was 1.5%)
channel muted by: 0 (was everybody)
# and the test that mattered: a real incident in November
22:14 CheckoutLatencyObjectiveBurning fires, pages
22:15 acknowledged
22:19 cause found via the runbook's first step
22:31 resolved
# under the old system the same incident would have
# produced 9 alerts in a muted channel.The action rate going from one and a half per cent to eighty-six is the measurement, and the November incident is the proof — a single page that woke the right person, with a runbook whose first step found the cause in four minutes.
What this costs
A gap somebody will point at after the next outage. Twenty-three deleted rules is twenty-three chances to say “we used to alert on that”, and the answer has to be the written record of what each would have caught — which is why the record exists.
The set will also grow back. Every incident produces pressure for a new rule, and the runbook requirement is the only thing standing against it — a requirement that is one exception away from being a formality, and the exception will be requested during an incident by somebody senior.