Four mechanisms built between 2023 and 2025, and the first incident where all of them were exercised at once.
04:12 systemd start limit → the unit fails
04:12 the absence alert on queue throughput fires
04:13 the alert routes to the channel, correctly
08:20 a human reads it, reverts the deploy
08:23 workers healthy
what each contributed:
the start limit stopped an infinite loop
the absence alert detected it. the error-rate
alerts did not, because there
were no errors — there was no
processing.
the routing put it in the right place
the rollback did nothing. it is bound to the
web health check.The absence alert is the one that did the work and it was added in 2025 as an afterthought to the integration monitoring. Alerting on a value not arriving is the category nobody adds, and it is the only thing that detects a component that has stopped rather than one that is failing.