Türker Yıldırım

monitoring

  • Two scheduled jobs nobody could account for

    Note1 min read

    A cron listing with eleven entries, of which two called commands that no longer existed. A cron job whose command does not exist fails silently unless something checks…

  • The test that had been quarantined since 2023

    Deep dive10 min read

    A quarantine list with a deadline, three extensions, and one test covering a payment path that had been skipped for three years.

  • A pipeline described in one file, after four years of five

    Deep dive10 min read

    Five workflow files, two composite actions, and nobody able to say what runs when. A change that needed editing three files, one of them wrong.

  • A drill that induced the same failure deliberately

    Note2 min read

    The April incident reproduced on purpose in May, to check whether the fix worked. Reproducing an incident deliberately is the only way to know a fix works, and…

  • Split-brain, and the check that refuses to promote

    Note1 min read

    A manual failover, and a guard against the case where the old primary is still writing. Refusing to promote while the old primary answers is the check that…

  • The certificate scan, two quarters on

    Note2 min read

    A quarterly scan of our own address space, after the first run found two things nobody owned. The first run of a scan like this finds the accumulated…

  • An event with no consumer, published for the last time

    Note1 min read

    An event dispatched on every order since 2021, subscribed to by a module that no longer exists. The monthly report had been correct since the module was discontinued…

  • The reporting database that is not a replica

    Deep dive10 min read

    A read replica serving reports, and reports that want a different schema. Eleven indexes on production tables that exist only for reporting.

  • DNS failover, and the resolver that ignored the TTL

    Note2 min read

    A sixty-second TTL, and traffic that took forty minutes to move. A sixty-second TTL produces four-minute failover for the great majority and a long tail that cannot be…

  • Building an MCP server for a database

    Deep dive15 min read

    turkerdev/mysql-mcp, and what a protocol for tool access actually requires. The authorisation model is the host’s problem, which means it is all yours.