monitoring – Page 21 – Türker Yıldırım

Compose v2 is a plugin, and the hyphen finally goes

Deep diveMay 19, 20219 min read

The same command, spelled two ways, on eight machines. A Python program became a Go plugin, and the container names changed underneath every script.

A correlation id header, agreed across two runtimes

NoteMay 19, 20212 min read

Two services in two languages each invented their own request identifier header, and neither could follow a request into the other. Deciding where the identifier originates is the…

The service that should not have been a service

Deep diveMay 14, 202110 min read

Four services, three of which deploy together and share a database. The costs of distribution with none of the independence.

Alert on the symptom, never on the cause

NoteMay 12, 20212 min read

An alert on a cause fires when nothing is wrong and stays quiet when something is, because the mapping from causes to consequences is not one to one.…

Replication lag measured with a heartbeat, not Seconds_Behind

NoteMay 11, 20212 min read

Seconds_Behind_Source is derived from the timestamp of the event being applied, so an idle source reports zero lag on a replica that is minutes behind. The failure that…

A histogram bucket cannot be changed retroactively

NoteMay 11, 20211 min read

Bucket boundaries are chosen before any data exists, and re-bucketing loses the history rather than recomputing it. Including the objective threshold as an explicit boundary is what makes…

SHOW REPLICA STATUS, and the terminology that changed

NoteApril 28, 20212 min read

8.0.22 deprecated the old replication vocabulary in favour of source and replica, and every script and runbook contains the old words. The column renames are what break monitoring…

The dotnet service that finally moved behind the same gateway

Deep diveApril 27, 202110 min read

One service in a different language, with its own auth, its own log format and its own deploy. The language was the least interesting difference.

An index that made writes slower than the reads it saved

Deep diveApril 23, 202110 min read

Eleven indexes on a table with a heavy write path. Every index is a write amplification that nobody measured, and three of them were never used.

Octane keeps the application in memory and finds every static you left

Deep diveApril 21, 202110 min read

40ms becomes 8ms, and the fourth request returns another customer’s data. The shared-nothing assumption is load-bearing in more places than anybody had counted.