deployment

  • An outage resolved before anybody woke up

    Four mechanisms built between 2023 and 2025, and the first incident where all of them were exercised at once. The absence alert is the one that did the…

  • A second host, and the first thing that was not replicated

    Everything in the provisioning script arrived on the new host, and three things that were not in it did not. A provisioning script describes configuration and a host…

  • Redis 8, a year on, and nothing to report

    A major version upgrade in May 2025, after thirteen months of a pin, and twelve months of nothing happening. Writing down that nothing happened is the only way…

  • The second host, and the failover we finally built

    Seven years on one host, and a business requirement that finally arrived. A recovery time that assumes the host comes back.

  • Decommissioning a supplier: the list

    An integration removed, and the eleven things that had to go with it. Eleven items and the code is one of them, which is the point — decommissioning…

  • nginx as the only thing holding a certificate

    A certificate loaded by one process, and a reload that had to work for the renewal to take effect. Three steps, of which the first two report success…

  • A path repository that reached staging

    A local path repository committed by accident, and an install on a machine where the path does not exist. A path repository is the right tool for developing…

  • Debian 13, and a provisioning script that needed nothing

    A major distribution upgrade, by rebuild, with no changes to the provisioning script at all. A provisioning script that is exercised monthly is a script that works when…

  • The integration that outlived the integrator

    A supplier acquired and shut down, and an endpoint returning 200 with an empty body for eleven weeks.

  • Automation that resolved a symptom and hid a cause

    A crash loop handled by a restart limit and an alert, which resolved the outage and delayed the diagnosis. Automatic recovery that stops a symptom without addressing the…