A threshold evaluated instantaneously fires on every transient spike, and transient spikes are constant — which is how a channel accumulates three hundred alerts a month and gets muted.
# fires on a single scrape. this is the noise.
- alert: CPUHigh
expr: node_load5 > 8
# a different claim: it has been true for ten minutes
- alert: SaturatedForTenMinutes
expr: node_load5 > 8
for: 10m
# and repeat_interval, so an ongoing incident is not re-paged
# every five minutes while somebody is working on it
The for clause is the single highest-value line in any rule and the one most often left out. repeat_interval is the other: an alert re-notifying every five minutes for an incident somebody is already handling trains that person to mute the channel, and the mute outlasts the incident. Both are one line and both are the difference between alerting and noise.