mysql

  • The second host, and the failover we finally built

    Seven years on one host, and a business requirement that finally arrived. A recovery time that assumes the host comes back.

  • A dual-write window, and the check that ran nightly

    Six weeks of writing two columns, and a check that they agree. The nightly agreement check is what makes a dual-write window safe, and it caught exactly the…

  • A retention report that asserts rather than describes

    A monthly report on retention that nobody read, converted into three assertions that fail. A report describes and an assertion fails, and the difference is whether anybody has…

  • An expand-and-contract for a column type

    A VARCHAR(32) becoming an ENUM, on a table with four hundred writes a second. Four releases for a column type is the honest cost of not being able…

  • Write latency, before and after eleven indexes

    Eleven reporting indexes dropped from production tables, measured on the write path. The median saving is trivial and the tail is not, which is the usual shape of…

  • A copy tool rejected because of a trigger

    An online schema change tool that works by adding triggers, on a table an outbox relay polls. The tool is well understood and the interaction with an existing…

  • A schema change on a table nobody can lock

    A column type change on a table taking four hundred writes a second, and an ALTER that estimates four hours.

  • An autoloaded option audit that found nothing

    A monthly check that has failed twice in two years, reporting clean for the fourteenth consecutive month. A check that has stopped failing is either a solved problem…

  • A restore from the second copy, timed

    A drill against the offsite copy rather than the primary, which took considerably longer for a reason nobody had costed. A recovery objective quoted from the fast path…

  • Reading from a replica, and the request that must not

    Two years of read splitting, and the twelfth place that had to read the primary. Naming the two reads differently is what keeps the count knowable — twelve…