Türker Yıldırım

monitoring

  • The registry we ran ourselves for three weeks

    Deep dive10 min read

    A rate limit hit at 09:00 every weekday, a pipeline failing intermittently for two months, and the wrong one of three options tried first.

  • The event nobody consumed, published for two years

    Note1 min read

    An event published on every order, with no subscribers since the service that listened was removed in 2022. Three events with no subscribers and one and a half…

  • An incident log read as an inventory rather than as history

    Note2 min read

    Three years of incident notes had been written for the record and never read as a data set. The last finding is the one worth acting on: the…

  • A runbook written by the person who does not know the answer

    Note2 min read

    Runbooks written by the expert were unusable by anybody else, so the second person wrote them and the expert reviewed. The expert’s version of step one was “check…

  • A systemd timer instead of a cron entry, at last

    Note2 min read

    A cron entry that had been failing silently for six weeks, and a timer that would have said so. Persistent=true runs a missed occurrence after a reboot, which…

  • Two caches with the same key and different meanings

    Note2 min read

    Two services writing settings:41, one meaning customer 41 and one meaning tenant 41, against the same instance. Separate logical databases on one instance are the other answer and…

  • The alert that woke the wrong person for a year

    Note2 min read

    A routing rule sending database alerts to a person who left the database team in 2023. A personal integration key survives a role change invisibly, because nothing about…

  • The service we merged back

    Deep dive14 min read

    Two services extracted in 2021, a team of four maintaining four pipelines, and forty-one of forty-four releases touching both.

  • Percentiles per consumer, not per endpoint

    Note2 min read

    An endpoint with a healthy p95 and one consumer for whom it was consistently slow. Aggregating across consumers averages away exactly the case that generates support tickets, and…

  • A shadowing month with the expert silent

    Note2 min read

    A month where the person who knows everything sat next to the person who does not and was not allowed to speak. The silence is the mechanism —…