monitoring – Page 15 – Türker Yıldırım

An admin panel that was a security boundary

Deep diveAugust 2, 202210 min read

/admin on the same hostname, behind a login form. A support engineer with the same session cookie as a customer.

Certificate expiry measured across the chain, not the leaf

NoteJuly 20, 20221 min read

Every expiry check looks at the leaf certificate, and a chain contains intermediates and roots with their own dates. earliest_cert_expiry is one word different from the metric everybody…

Log sampling, for the line that fires ten thousand times

NoteJuly 5, 20222 min read

A warning that fires on every request during an incident is the majority of the log volume and adds nothing after the first hundred. Recording the sample rate…

A GDPR deletion request across eleven tables

Deep diveJuly 4, 202214 min read

A request arrived and nobody could say where the data was. Personal data in eleven tables, three of them denormalised copies.

Sessions in Redis, and the logout that did not

Deep diveJune 29, 202210 min read

A session store move that changed the security properties. Logging out on one device leaves the others signed in.

A recording rule is a query you stopped rewriting

NoteJune 9, 20221 min read

The same ratio appears in a dashboard, two alerts and a report, and each copy can drift from the others. A naming convention with colons is the established…

The deploy that needed a database restore

Deep diveJune 7, 202215 min read

A migration that wrote bad data to 41,000 rows. Rollback restores the code and not the rows.

The queue table that became a bottleneck at four million rows

Deep diveJune 3, 202210 min read

A database-backed queue that worked for four years. SELECT FOR UPDATE SKIP LOCKED, and a lock wait timeout.

An error budget spent deliberately is the point of having one

NoteMay 31, 20222 min read

A budget that is never spent means the objective is too low, and a budget that is exhausted every month means it is too high or the system…

The report that ran on a replica and was three minutes stale

Deep diveMay 31, 202210 min read

Read routing added for capacity, and a report that disagreed with the dashboard. Two numbers, both correct, three minutes apart.