monitoring

  • Multi-window burn rate, and the two alerts it replaces

    A single burn-rate window is either too twitchy or too slow, and pairing a long window with a short one gives both sensitivity and stability. The short window…

  • ANALYZE TABLE, and statistics that were three months old

    The optimiser chooses a plan from statistics it samples, and a table whose distribution has changed since the last sample gets a plan chosen for the old data.…

  • Log sampling, because volume is a line on an invoice

    Logging every request at info level produces a volume nobody reads and a bill somebody notices, and the useful lines are a small fraction of it. Recording the…

  • Draining a node on a Wednesday afternoon, on purpose

    A failover that has never been exercised is a procedure nobody has tested, and the first execution is always during an incident with an audience. Every one of…

  • Apdex, and a number somebody outside engineering can read

    A p99 in milliseconds means nothing to somebody who does not think in latency distributions, and Apdex compresses the same data into a number between zero and one.…

  • systemd-run for a one-off with a memory limit

    Running a large import by hand and taking the machine down with the OOM killer is a recurring way to turn a routine task into an incident. –scope…

  • Seconds_Behind_Master lies during a long ALTER

    The three status fields people read all measure something slightly different from what the question was, and knowing which is which is the difference between a useful alert…

  • A trace is a tree; a log line is a point

    Logs answer what happened and traces answer what a request did, and a request that touches six services is a shape logs cannot represent. The value is entirely…

  • Health checks are middleware, not a controller

    A health endpoint written as a controller action goes through routing, filters and model binding — which is exactly the machinery a health check should not depend on.…

  • Draining a node before deploying to it

    Deploying to a host still receiving traffic means the requests in flight during the swap see a half-updated application, and there is no version of that which is…