Three years of incident notes, read as an inventory rather than as history. Eight documents cover eighty-seven per cent of what has ever gone wrong, which is a…
A renewal job that had been running successfully for two years, and a certificate that had been served from a stale copy for the last three months. Monitoring…
Three attempts, one second apart, applied to every consumer regardless of what it does. One retry policy across four consumers means three of them are wrong, and the…
Full tracing on a service doing forty thousand requests a day is a data volume problem; sampling uniformly loses the requests worth looking at. Head sampling decides at…
A message that cannot be processed and must not be skipped, in a stream where order matters. Halting is the right answer when skipping would corrupt the state…
Eight severity levels, used by four people with four different interpretations, producing a log where error meant nothing in particular. The column that resolved the arguments is the…
A deploy that succeeds because the files copied is a deploy that has verified nothing. Twenty seconds of retrying before giving up, and a rollback that is the…
Replication lag of eleven minutes on a replica with no load, caused by a single transaction on the primary that took eleven minutes. The instant drop is the…