A crash loop handled by a restart limit and an alert, which resolved the outage and delayed the diagnosis.
what the automation did:
stopped the loop, alerted, and left the workers down
what it did not do:
say why, or roll back the deploy that caused it
the deploy at 03:58 was the cause and the deploy's own
health check passed, because it checks the web tier.
so the automation resolved the visible failure and the
broken release stayed live until 08:20, when somebody
read the alert and reverted it.Automatic recovery that stops a symptom without addressing the cause converts an outage into a longer degraded period, which is usually better and is not obviously better. Extending the deploy health check to cover the workers is the fix and it makes the check slower and more likely to produce a false rollback.