logging

  • A container that logged to stdout and to a file

    An application writing to both, which doubles the disk and halves the usefulness of each. A container writing to a file is a container whose logs disappear when…

  • A post-incident note written by somebody who slept through it

    An incident resolved automatically overnight, written up from the timeline rather than from memory. Writing it up from logs rather than from experience produced a better note than…

  • An alert into a channel nobody reads at 04:00

    An alert that fired correctly at four in the morning, into a channel that is read at eight. An alert that is correct, correctly routed and unread for…

  • The incident that was a certificate, again

    The third certificate incident in six years, each with a different cause. An expired certificate on a service nobody remembered was serving TLS.

  • Ten years of a codebase, read in one week

    A decade of an application, and a week spent reading it rather than changing it. Decisions re-litigated because nobody knows what is already there.

  • json_encode with JSON_THROW_ON_ERROR, everywhere, finally

    A codebase-wide sweep for json_encode and json_decode without the throw flag, eleven years after the flag arrived. Twenty-two silent failures is the finding, and one of them had…

  • An alert deleted after three months of silence

    A rule that had not fired since it was created, deleted on the grounds that a silent alert is untested rather than reassuring. A rule that never fires…

  • The PSR-18 client I wrote for one API

    A supplier API with a signing scheme, a rate limit and a pagination style of its own, and the same four concerns implemented in three places.

  • The endpoint that was slow for one customer

    A healthy p95 and one customer for whom the endpoint takes four seconds. An aggregate metric that averages away the only case that matters.

  • The runbook that was wrong because the command changed

    A runbook step naming a command that had been renamed in a refactor eight months earlier. A runbook is documentation that must execute correctly under pressure, which makes…