monitoring – Page 10 – Türker Yıldırım

A replica that fell behind because of one long transaction

NoteOctober 31, 20232 min read

Replication lag of eleven minutes on a replica with no load, caused by a single transaction on the primary that took eleven minutes. The instant drop is the…

The bundle we split by route, and the waterfall it created

Deep diveOctober 24, 202310 min read

A 900 KB bundle, split by route, and a page that got slower. Six sequential requests before the first render.

Healthchecks that check the thing, not the port

NoteOctober 24, 20232 min read

A container health check that opened a TCP connection to 9000 and reported healthy while every request returned a 500. A port check answers “did the process start”,…

Journald rate limiting, and the log line that vanished

NoteOctober 19, 20231 min read

A worker logging a line per job hit the journal’s rate limit during a backlog, and the lines that were dropped were the ones describing the backlog. The…

The alert that fired for two years and meant nothing

Deep diveOctober 18, 202310 min read

Thirty-one alert rules, a channel muted by everybody, and 412 alerts in a month of which four corresponded to a real problem.

Import blocks, and the resources we had been pretending to manage

Deep diveSeptember 21, 202310 min read

Terraform 1.5 in June, and an import that is part of the plan rather than a side effect. Fourteen resources created by hand in 2019.

An outbox, and the publisher we removed from the transaction

Deep diveSeptember 20, 202310 min read

An order committed and an event never published, twice in one week. A broker call inside a database transaction, and a rollback after it.

Deleting thirty alerts, and the one that mattered

NoteSeptember 5, 20231 min read

Thirty-one rules, 412 firings in a month, four of which corresponded to something a user would have noticed. The rule that survived unchanged is the one written last…

An event log that is not an event store

Deep diveAugust 10, 202310 min read

Wanting to know what happened to an order without adopting event sourcing. A support question that takes twenty minutes and three tables.

The N+1 that only appeared in production

Deep diveJuly 29, 202310 min read

An endpoint at 40 ms locally and 2.1 seconds in production. Nine hundred queries on one request, and three on the same request in a test.