monitoring

  • A postmortem without an owner is a document

    The value of a postmortem is entirely in the actions, and an action without a name and a date is a wish. Reviewing the action list from the…

  • Escalating is not failing, and the runbook should say so

    A rotation where asking for help feels like failure produces two hours of struggling on something the expert fixes in ten minutes. Putting it in writing, in every…

  • A CSV export that streams instead of collapsing

    An export that worked until the account with 400,000 rows. fetchAll, a collection and a string built in memory, at 512 MB.

  • Sampling traces at 1% misses the request you care about

    Head-based sampling decides before the request runs, so the eight-second outlier is in the sample only by luck. Tail-based sampling is the correct answer and needs infrastructure that…

  • docker system df, before you buy more disk

    A build host filling its disk is almost always reclaimable space rather than a capacity problem, and the breakdown says which kind. The build cache is usually the…

  • Time-to-start per queue, as a metric

    Queue depth without throughput says nothing: a hundred jobs is fine at fifty a second and an incident at one a minute. Oldest-waiting-age is a direct measure of…

  • A healthcheck that tests the app, not the port

    A health check that opens a TCP connection reports healthy for a process that is listening and cannot serve a request. The rule that has held up is…

  • Burn rate beats a threshold on the raw metric

    A threshold on an error rate fires at the same point whether the budget is fresh or already spent, which is not the decision anybody wants to make.…

  • Queue priority is not a number, it is a separate queue

    Password resets stuck behind a nightly export. A priority field does not help when the worker is already four minutes into something else.

  • A webhook endpoint that survives the sender

    The payment provider retried for six hours and every attempt timed out. A webhook is a notification, not a request for processing.