Türker Yıldırım

deployment

  • A cold start after a failover

    Note1 min read

    A promoted standby serving traffic with an empty cache and a cold opcode cache, measured during a drill. The warm-up script existed and was bound to the wrong…

  • WP-CLI in the failover runbook

    Note1 min read

    A failover procedure with three WP-CLI steps, and the one that has to run before the database is writable. Three commands and an ordering constraint that is not…

  • Two scheduled jobs nobody could account for

    Note1 min read

    A cron listing with eleven entries, of which two called commands that no longer existed. A cron job whose command does not exist fails silently unless something checks…

  • Deleting a module

    Deep dive10 min read

    A module built in 2022 for a product line discontinued in 2024. Eight thousand lines, a database schema and two scheduled jobs, all still running.

  • A pipeline described in one file, after four years of five

    Deep dive10 min read

    Five workflow files, two composite actions, and nobody able to say what runs when. A change that needed editing three files, one of them wrong.

  • A drill that induced the same failure deliberately

    Note2 min read

    The April incident reproduced on purpose in May, to check whether the fix worked. Reproducing an incident deliberately is the only way to know a fix works, and…

  • Split-brain, and the check that refuses to promote

    Note1 min read

    A manual failover, and a guard against the case where the old primary is still writing. Refusing to promote while the old primary answers is the check that…

  • The reporting database that is not a replica

    Deep dive10 min read

    A read replica serving reports, and reports that want a different schema. Eleven indexes on production tables that exist only for reporting.

  • DNS failover, and the resolver that ignored the TTL

    Note2 min read

    A sixty-second TTL, and traffic that took forty minutes to move. A sixty-second TTL produces four-minute failover for the great majority and a long tail that cannot be…

  • A shared object store, and the session problem it did not solve

    Note2 min read

    Uploads moved to shared storage, and sessions still tied to whichever host served the previous request. The session store was shared and the cookie was not, which is…