monitoring – Page 31 – Türker Yıldırım

A dead letter queue with no alert is a bin

NoteAugust 6, 20192 min read

Routing failed messages somewhere they cannot block the main queue is correct design, and it converts a loud failure into a silent one unless something watches the depth.…

Structured logs mean the message is a constant

NoteJuly 31, 20192 min read

Interpolating values into the message produces a unique string per event, so an aggregator sees a million distinct messages rather than one with a million occurrences. The message…

The queue worker that leaked, and the memory limit that hid it

Deep diveJuly 15, 201910 min read

A worker restarting every eleven minutes, reported as healthy. Finding the leak is not the same as bounding it, and the seatbelt hides the next one.

systemd-run for a one-off command with a memory limit

NoteJuly 11, 20191 min read

Running a large import by hand and having it take the machine down with the OOM killer is a recurring way to turn a routine task into an…

Rootless containers and the build that stopped needing a daemon

Deep diveJuly 4, 20199 min read

19.03 made BuildKit the default and rootless mode possible. A CI runner where building an image means root on the host is the problem worth naming.

Messenger, and the queue that is part of the framework

Deep diveJune 25, 201910 min read

4.3 made Messenger stable, and dispatch stopped being the same decision as transport. A message, a handler, and routing declared in configuration.

Lazy collections and the report that would not fit in memory

Deep diveJune 24, 201910 min read

A memory limit hit on the fourth of the month, every month. A collection is an array, and an array is all of it at once — until…

predict_linear for a disk, instead of a percentage threshold

NoteJune 11, 20191 min read

Eighty percent on a disk growing a percent a quarter is not a problem, and eighty percent on a disk that gained fifteen points overnight is — and…

Draining a node before you deploy to it

NoteJune 4, 20192 min read

Deploying to a host that is still receiving traffic means the requests in flight during the swap see a half-updated application, and there is no version of that…

A sampling profiler is the one that belongs in production

NoteMay 28, 20192 min read

An instrumenting profiler records every function call, which changes the timings it is measuring — small fast functions look disproportionately expensive because the overhead is per call. Sampling…