A rule that had not fired since it was created, deleted on the grounds that a silent alert is untested rather than reassuring.
the rule: replica lag above 60 seconds for 5 minutes.
fired: never, in 14 months.
the question: is the replica healthy, or is the rule
broken?
the test: pause replication on the replica for six
minutes, deliberately, on a Tuesday.
the result: the rule did not fire. the metric had been
absent since the exporter was upgraded in March, and
an expression over an absent metric evaluates to
nothing rather than to a breach.A rule that never fires is indistinguishable from a rule that cannot fire, and the only way to tell is to cause the condition. The absent-metric case is the general hazard with any threshold expression and the guard is a companion rule using absent(), which is now a required companion for every threshold alert.