monitoring – Page 11 – Türker Yıldırım

Structured logs with a correlation id, end to end

NoteJuly 25, 20232 min read

A request that touches the web process, a queue job and an outbound API call produces three sets of logs with nothing connecting them. The propagation across the…

The state file two people wrote to at once

Deep diveJuly 14, 202310 min read

Two pipelines, one state file, and a plan that proposed deleting a database. A lock held by a run cancelled forty minutes earlier.

A certificate renewal that failed silently for sixty days

NoteJune 30, 20231 min read

Automated renewal that had been failing since a path changed, discovered with nine days to spare by somebody looking for something else. Monitoring the renewal job tells you…

Partitioning by month, and the query that scanned all of them

Deep diveJune 27, 202310 min read

A 400 million row events table and a delete that could not finish. Dropping a partition takes two seconds; getting there takes a schema change.

OpenTelemetry in PHP, and auto-instrumentation that was not

NoteJune 7, 20232 min read

The PHP SDK reached 1.0 this year and the auto-instrumentation is thinner than the equivalent in other runtimes. The gap is the whole problem — a trace that…

Two years of deprecations, and the upgrade we could no longer defer

Deep diveJune 6, 202314 min read

PHP 8.0 reaches end of life in November and the application is on it. 4,100 deprecation notices, and a log filter that had been hiding them.

An error budget, and the deploy freeze we did not need

NoteMay 30, 20232 min read

The policy said a burnt budget triggers a deploy freeze; the budget was burnt by one incident and the freeze would have blocked the fix. A freeze assumes…

Rebuilding the deploy as something a stranger could run

Deep diveMay 16, 202314 min read

A deploy that one person can run, and a holiday booked for August. A runbook of 22 steps, four of which say “check it looks right”.

The anti-corruption layer we should have written first

Deep diveMay 9, 202310 min read

A supplier who renamed four fields in a minor release, without notice, and twenty-two files touched to accommodate it.

opcache statistics, and the restart that cost 400 ms

NoteApril 27, 20232 min read

A cache that fills, restarts, and refills, with every restart being a period where every request compiles everything. The restart counters are the numbers to alert on and…