deployment

  • A schema change on a table nobody can lock

    A column type change on a table taking four hundred writes a second, and an ALTER that estimates four hours.

  • A runbook command check that caught three

    A weekly check that every command named in a runbook still exists, which found three after a refactor. A runbook is documentation that must execute correctly under pressure,…

  • A systemd unit with a restart limit that finally fired

    A burst limit configured in 2023 as a precaution, which took three years to be needed. Without the burst limit the worker would have restarted every five seconds…

  • A restart limit that fired for the first time

    A systemd burst limit configured in 2023 as a precaution, which took three years to be needed. Without the limit the worker would have restarted every five seconds…

  • A restore from the second copy, timed

    A drill against the offsite copy rather than the primary, which took considerably longer for a reason nobody had costed. A recovery objective quoted from the fast path…

  • PHP 8.1 reaches end of life, and the one host still on it

    December, the end of security support for 8.1, and a host running it that nothing in the deployment inventory knew about. The end-of-life date is not the story…

  • A second backup provider with write-only credentials

    Every backup in one provider, and a plan that assumed the disaster would be local rather than at the provider. Ranking the failure modes is what makes this…

  • A plugin with no admin screen

    A plugin whose entire configuration is a constant and a filter, on a site whose deployment is a pipeline.

  • A single binary against a runtime, costed

    A recurring proposal to move a service to a compiled language, costed properly rather than argued about. The performance difference is real and is irrelevant to the caller,…

  • The backup that had to leave the building

    Every backup in one provider’s object storage, and a disaster recovery plan that assumes the disaster is local.