monitoring

  • Percentiles labelled by consumer, and the outlier

    One label on a histogram, and a p95 that had been describing nobody. An aggregate percentile is a weighted average of populations that may have nothing in common,…

  • The PSR-18 client I wrote for one API

    A supplier API with a signing scheme, a rate limit and a pagination style of its own, and the same four concerns implemented in three places.

  • A dead letter queue that reached zero

    A dead letter queue that had held between forty and eight thousand messages since 2019, empty for the first time. An empty dead letter queue is not the…

  • Two years of query logs, read once

    A slow query log rotating for two years and never aggregated, and performance work driven by whoever complained loudest.

  • A registry cache that filled a disk on a Sunday

    A pull-through cache with no garbage collection schedule, and a disk that filled at two in the morning. A cache grows without bound by definition and the only…

  • The endpoint that was slow for one customer

    A healthy p95 and one customer for whom the endpoint takes four seconds. An aggregate metric that averages away the only case that matters.

  • The runbook that was wrong because the command changed

    A runbook step naming a command that had been renamed in a refactor eight months earlier. A runbook is documentation that must execute correctly under pressure, which makes…

  • A licence that came back, and the migration we had prepared

    The first of May, an AGPL release, and a decision from April 2024 that can finally be closed — in the direction nobody planned for.

  • The cron entry that outlived three rewrites of what it called

    A crontab line from 2019 calling a script that had been rewritten twice and moved once, still working because of a symlink nobody remembered. Every layer was added…

  • A dashboard rebuilt from the questions people ask

    A dashboard of fourteen panels, replaced by six, chosen by reading a quarter of incident messages. A dashboard built from available metrics answers what is measurable and a…