Türker Yıldırım

monitoring

  • A shadowing month with the expert silent

    Note2 min read

    A month where the person who knows everything sat next to the person who does not and was not allowed to speak. The silence is the mechanism —…

  • Eight failure modes covering forty-one incidents

    Note2 min read

    Three years of incident notes, read as an inventory rather than as history. Eight documents cover eighty-seven per cent of what has ever gone wrong, which is a…

  • The certificate that renewed itself and nobody verified

    Note2 min read

    A renewal job that had been running successfully for two years, and a certificate that had been served from a stale copy for the last three months. Monitoring…

  • A retry policy copied from a tutorial in 2019

    Note2 min read

    Three attempts, one second apart, applied to every consumer regardless of what it does. One retry policy across four consumers means three of them are wrong, and the…

  • Reproducing a build from six months ago

    Deep dive14 min read

    A security question about a release from July, and no way to rebuild it. The same commit produces a different image, and neither of them is wrong.

  • Sampling traces at five per cent, keeping every error

    Note2 min read

    Full tracing on a service doing forty thousand requests a day is a data volume problem; sampling uniformly loses the requests worth looking at. Head sampling decides at…

  • A consumer that stopped on a poison message, on purpose

    Note2 min read

    A message that cannot be processed and must not be skipped, in a stream where order matters. Halting is the right answer when skipping would corrupt the state…

  • The unique index we could not add, and the duplicates we had to find first

    Deep dive10 min read

    A constraint that should have existed since 2016, and 1,240 rows that violate it. Support merging customer records by hand, monthly.

  • Log levels agreed in writing, finally

    Note1 min read

    Eight severity levels, used by four people with four different interpretations, producing a log where error meant nothing in particular. The column that resolved the arguments is the…

  • A deploy that rolls back on a health check

    Note2 min read

    A deploy that succeeds because the files copied is a deploy that has verified nothing. Twenty seconds of retrying before giving up, and a rollback that is the…