A threshold evaluated instantaneously fires on every transient spike, and transient spikes are constant — which is how a channel accumulates three hundred alerts a month.
# fires on a single scrape
- alert: CPUHigh
expr: node_load5 > 8
# a different claim: it has been true for ten minutes
- alert: SaturatedForTenMinutes
expr: node_load5 > 8
for: 10m
# and repeat_interval, so an ongoing incident is not re-paged
# every five minutes at somebody already working on it
The for clause is the single highest-value line in any rule and the one most often left out. repeat_interval is the other: re-notifying every five minutes trains the recipient to mute the channel, and the mute outlasts the incident. Both are one line each and both are the difference between alerting and noise.