linux

  • A host rebuilt from the provisioning script, for the first time

    A provisioning script maintained since 2019 and never once run against an empty machine. A script that only ever runs against an already-working host converges to describing that…

  • The shell script that was rewritten and stayed shell

    A six-hundred-line deploy script rewritten in a compiled language in 2024, and a second rewrite back to shell. The rewrite was correct about the problem and wrong about…

  • An on-call rota of three

    A rota of two is not a rota. A third person joined in November, and the gap was everything learned in the shadowing month and never written down.

  • nginx 1.26, and a config that needed nothing

    The stable branch moved in April and the configuration required no changes, which is worth writing down as often as the opposite. The http2 on directive is the…

  • Ubuntu 24.04, and the two packages that moved

    A rebuild onto the new long-term-support release, which was uneventful except for two packages that had been renamed. The two package errors stopped the build, which is the…

  • Handing over on-call to one other person

    One person who can fix anything, and a rota of one. Three years of incidents resolved by the same person, every time.

  • A licence change, and the fork we did not take

    20 March, a licence change, and a fork announced within a week. A dependency that is no longer open source, in production, everywhere.

  • A shell script that should have been a program

    A deployment script that had grown to six hundred lines of bash with functions, associative arrays and a retry loop. Bash is the right tool for orchestrating commands…

  • The registry we ran ourselves for three weeks

    A rate limit hit at 09:00 every weekday, a pipeline failing intermittently for two months, and the wrong one of three options tried first.

  • An ssh certificate instead of an authorized_keys file

    Adding a person meant editing a file on eleven hosts, and removing one meant remembering to. The eight-hour validity is what makes revocation mostly unnecessary — a departed…