monitoring – Page 13 – Türker Yıldırım

Larger runners, and the pipeline that was memory-bound

Deep diveDecember 13, 202210 min read

October, larger runners, and a build that was swapping. A Node build that used 6 GB on a 7 GB runner.

The webhook we sent, and the retries we owed

Deep diveDecember 7, 202210 min read

Being the sender, for the first time, and the obligations that come with it. A customer endpoint that was down for six hours.

A pipeline that only builds what changed

Deep diveNovember 15, 202210 min read

A 22-minute pipeline on a repository where most changes touch one directory. Every push runs everything, including the untouched.

A queue worker that scales on the metric that matters

Deep diveNovember 9, 202210 min read

Six workers, always, whether there is work or not. A backlog at 09:00 and idle capacity at 03:00.

An idempotent consumer for a broker that redelivers

Deep diveNovember 8, 202210 min read

A message processed twice, and a customer charged twice. At-least-once delivery, treated as exactly-once by the handler.

Two teams, one database, and the schema change that needed a meeting

Deep diveOctober 29, 202210 min read

A shared schema, and a rename that took three weeks to coordinate. Table ownership that exists in nobody’s head.

Two burn rates: one pages, one raises a ticket

NoteOctober 28, 20222 min read

A single burn-rate alert either pages for a slow leak or misses a fast one, and two thresholds cover both without either being wrong. The multipliers come from…

An SLI is a ratio of good events to valid events

NoteOctober 18, 20221 min read

Both halves of the definition are decisions, and the denominator is the one people forget to make. Excluding 4xx from the denominator is defensible and is a decision…

The objective is a negotiation, not an aspiration

NoteOctober 18, 20222 min read

A number chosen by engineering alone is a number the business has not agreed to constrain the roadmap with, which is what an objective is for. Presenting it…

A cache invalidation bug that only appeared under load

Deep diveOctober 17, 202210 min read

Correct in every test, wrong for about one request in four thousand. A read that races a write, and a cache that keeps the loser.