Türker Yıldırım

monitoring

  • A queue table is not a queue

    Deep dive10 min read

    SELECT … WHERE processed = 0 LIMIT 1, and two workers taking the same row. SKIP LOCKED makes the table viable; it does not make it a broker.

  • Monit restarting a process is a symptom, so alert on the restart

    Note2 min read

    A process supervisor that quietly restarts a service is doing its job and hiding the problem, so a memory leak becomes a daily restart nobody knows about. The…

  • Wildcard certificates, and the DNS challenge

    Deep dive9 min read

    ACME v2 landed in March and wildcards need a different proof. The certificate is the easy part; the API token that can rewrite your zone is not.

  • A swap file on a small VPS, and the OOM killer it prevents

    Note1 min read

    A one-gigabyte VPS running MySQL, PHP and nginx has no headroom, and the kernel resolves that by killing the largest process — which is MySQL, mid-write. Swappiness at…

  • A backup you have not restored is not a backup

    Note2 min read

    A nightly dump that has run successfully for two years proves that the dump command exits zero, which is a different claim from the data being recoverable. The…

  • Laravel 5.6 and logging you can configure

    Deep dive10 min read

    Logging moved into a config file, and stacks are why it matters — a different format per destination, decided without touching a service provider.

  • Measure the p99, because the average hides the incident

    Note2 min read

    An hourly spike that made every request four hundred milliseconds slower for two seconds moved the average by nine milliseconds and was invisible in every graph anyone looks…

  • A circuit breaker before the third-party outage

    Deep dive10 min read

    A payment provider was slow and the whole site went down with it. A timeout bounds one request; it does nothing about how many are waiting at once.

  • journald rate limiting drops your logs silently

    Note1 min read

    systemd-journald discards messages from a service exceeding its burst limit and notes the fact in a single line that is easy to miss — so a debugging session…

  • Alerting that people do not learn to ignore

    Deep dive10 min read

    340 alerts in a month, four of which mattered. An alert channel nobody reads is the same as no alerting, and the cause is alerting on causes.