backup

  • An audit log that a lawyer could read

    A dispute, and a log that recorded that something changed. “Order updated” with no before, no after and no actor.

  • data_free, and the 80% of the file that is nothing

    InnoDB reuses freed space within a table and does not return it to the filesystem, so a table deleted from daily can be mostly empty and full size.…

  • The on-call rotation for a team of four

    One person had the phone for two years. A rotation of four is a week in four, which is enough to be tiring and not enough to build…

  • The migration that squashed two hundred files

    A test suite spending ninety seconds replaying six years of schema history. Squashing is a good idea with two failure modes worth knowing first.

  • Secrets that are not in the repository and not in the image

    The database password was in the repository, then in the image, then in an environment variable readable by anything that could run ps. Where it went next.

  • Auto-updates, and the site that changed underneath you

    5.5 turns on plugin auto-updates per plugin, and a managed site is not a hobby site. A plugin updated at 03:00 and a checkout broken at 09:00.

  • predict_linear for a disk, not a percentage

    Eighty percent on a disk growing a percent a quarter is not a problem, and eighty percent on one that gained fifteen points overnight is — a single…

  • An SLO is a number somebody agreed to

    Everyone wanted the site to be fast and nobody could say what fast meant. An indicator, an objective and a window — and the error budget is what…

  • Scanning history for a leaked key, and what to do next

    A secret committed and then removed is still in the history, and the history is on every clone, every fork and every CI cache. Assuming the secret is…

  • A for clause on every rule, or it fires on noise

    A threshold evaluated instantaneously fires on every transient spike, and transient spikes are constant — which is how a channel accumulates three hundred alerts a month. The for…