Türker Yıldırım

monitoring

  • A status page nobody had ever updated during an incident

    Note2 min read

    A status page maintained since 2022, updated eleven times, none of them during an actual outage. Putting the communication before the diagnosis feels wrong and is right —…

  • A variable that was removed rather than deprecated

    Note2 min read

    A configuration file carried forward since 2018, containing a variable the new version refuses to start with. A refusal to start is the correct severity for an unknown…

  • A cache that was a second database

    Deep dive10 min read

    A cache holding data that existed nowhere else, discovered during a restart. A flush that lost four hours of user preferences.

  • An API that had two clients and three shapes

    Deep dive10 min read

    One endpoint, two consumers, and a response that differs by header. Three serialisation paths for one resource, two of them accidental.

  • Handing over on-call to one other person

    Deep dive10 min read

    One person who can fix anything, and a rota of one. Three years of incidents resolved by the same person, every time.

  • Health routing, and the check we already had

    Note2 min read

    A framework health route that returns 200 if the application boots, next to ours that checks the database and the cache. These answer different questions and using one…

  • An outbox relay that stalled and told nobody

    Note2 min read

    A relay process that died on an unhandled exception and left rows accumulating with no error anywhere. The second alert is the one that mattered: the relay stopped…

  • A licence change, and the fork we did not take

    Deep dive10 min read

    20 March, a licence change, and a fork announced within a week. A dependency that is no longer open source, in production, everywhere.

  • A runner that ran out of disk halfway through a build

    Note2 min read

    A hosted runner has about fourteen gigabytes free, and a Docker build with a large cache uses all of it without saying so. The preinstalled toolchains for languages…

  • MySQL 8.4, and the defaults that changed underneath us

    Deep dive10 min read

    The first long-term-support release under the new model, and an application that could not authenticate after a container image bump.