elk-stack

  • The on-call rotation for a team of four

    One person had the phone for two years. A rotation of four is a week in four, which is enough to be tiring and not enough to build…

  • Observability is three things, and logs are one

    Forty gigabytes of logs a day and no answer to “is checkout slow”. Metrics, logs and traces answer different questions, and logs were doing all three badly.

  • An SLO is a number somebody agreed to

    Everyone wanted the site to be fast and nobody could say what fast meant. An indicator, an objective and a window — and the error budget is what…

  • Log sampling, because volume is a line on an invoice

    Logging every request at info level produces a volume nobody reads and a bill somebody notices, and the useful lines are a small fraction of it. Recording the…

  • Redaction belongs in the pipeline, not the application

    Removing personal data at the call site catches what you thought about; removing it at the shipper catches the debug line somebody adds next year. The regular expression…

  • Fatal errors that reach a person before they reach a customer

    The error tracker had 400,000 events and nobody had opened it in a month. Alert fatigue, in the one tool that exists to prevent it.

  • Redaction belongs in the shipping pipeline, not the application

    Removing personal data at the call site catches what you thought about; removing it at the shipper catches the debug line somebody adds next year in a code…

  • Structured logs mean the message is a constant

    Interpolating values into the message produces a unique string per event, so an aggregator sees a million distinct messages rather than one with a million occurrences. The message…

  • Elasticsearch 7 removed mapping types

    One type per index in 6, none at all in 7, and every query URL changes. The reindex cannot be done in place — and the default shard…

  • Deleting a user is harder than creating one

    The delete request arrived and the row was the easy part. Foreign keys model structure, not ownership — and the person is in six other places.