A crash loop handled by a restart limit and an alert, which resolved the outage and delayed the diagnosis. Automatic recovery that stops a symptom without addressing the…
One label added in 2025, still the thing that surfaces problems no aggregate shows. A consumer at four times the aggregate is a signal that no absolute threshold…
A supplier endpoint answering successfully with an empty body, for eleven weeks, after the supplier was acquired. A successful response with no content is indistinguishable from a successful…
An application writing to both, which doubles the disk and halves the usefulness of each. A container writing to a file is a container whose logs disappear when…
A weekly check that every command named in a runbook still exists, which found three after a refactor. A runbook is documentation that must execute correctly under pressure,…
A burst limit configured in 2023 as a precaution, which took three years to be needed. Without the burst limit the worker would have restarted every five seconds…
A systemd burst limit configured in 2023 as a precaution, which took three years to be needed. Without the limit the worker would have restarted every five seconds…
An incident resolved automatically overnight, written up from the timeline rather than from memory. Writing it up from logs rather than from experience produced a better note than…