Türker Yıldırım

monitoring

  • Compose in CI, and the service the tests need

    Deep dive10 min read

    The tests need MySQL and Redis, and CI gives you an empty machine. Service containers, where they differ from compose, and the readiness nothing does for you.

  • predict_linear for a disk, not a percentage

    Note1 min read

    Eighty percent on a disk growing a percent a quarter is not a problem, and eighty percent on one that gained fifteen points overnight is — a single…

  • A read replica that was silently stale

    Deep dive15 min read

    The replica was up, accepting connections and serving data from an hour ago. Seconds_Behind_Master reported zero, honestly, and was measuring the wrong thing.

  • journalctl -o json is a log pipeline with no agent

    Note1 min read

    The journal already stores structured records with a unit, a priority and a hostname; the text output is a rendering rather than the data. Following the journal in…

  • Sampling, because tracing every request is not affordable

    Note2 min read

    Full-fidelity tracing on a busy service is a considerable volume of data and a considerable bill, and almost all of it describes requests that were fine. Head-based sampling…

  • A self-hosted runner, and what it is allowed to reach

    Deep dive10 min read

    The hosted runner cannot reach the private network, which is the whole problem — and a runner inside it is a machine running other people’s code.

  • An SLO is a number somebody agreed to

    Deep dive14 min read

    Everyone wanted the site to be fast and nobody could say what fast meant. An indicator, an objective and a window — and the error budget is what…

  • A canary needs a metric to decide on, agreed in advance

    Note2 min read

    A canary release without a decision rule is a slower deploy — somebody watches a dashboard for ten minutes, sees nothing obvious and promotes it. The minimum sample…

  • The error budget is what the objective is for

    Note2 min read

    An objective of 99.5% is also a statement that 0.5% of requests may fail, and that allowance is a resource with a use. The budget turns reliability from…

  • Context propagation is the whole of distributed tracing

    Note2 min read

    Instrumenting each service produces disconnected traces. Passing the trace context between them is what makes it one trace, and it is the part that breaks. The queue hop…