An alert that fired correctly at four in the morning, into a channel that is read at eight.
the routing, as designed:
page the service cannot serve, or money is
being lost. 3 rules.
channel somebody looks during working hours.
5 rules.
the rule that fired: a worker crash loop, which is in
the channel category because the automatic recovery
handles it.
and it did. the alert was correct, the routing was
correct, and it sat unread for four hours while nothing
was wrong.An alert that is correct, correctly routed and unread for four hours is the system working, which is counter-intuitive enough to be worth writing down. The only thing that would have improved the outcome is a human being awake, and the automation had already resolved the symptom.