deployment – Page 12 – Türker Yıldırım

The alert that fired for two years and meant nothing

Deep diveOctober 18, 202310 min read

Thirty-one alert rules, a channel muted by everybody, and 412 alerts in a month of which four corresponded to a real problem.

Import blocks, and the resources we had been pretending to manage

Deep diveSeptember 21, 202310 min read

Terraform 1.5 in June, and an import that is part of the plan rather than a side effect. Fourteen resources created by hand in 2019.

Debian 12 on a host that had been on 10

NoteAugust 8, 20232 min read

Two major versions in one jump, which the documentation says not to do and which is what the situation actually was. A rebuild is the honest test of…

A remote state lock held by a run that was cancelled

NoteAugust 5, 20232 min read

A pipeline cancelled during an apply leaves the lock held, and every subsequent run fails with a lock ID and a timestamp. Step two is the one people…

check blocks, and the assertion that runs after apply

NoteJuly 26, 20232 min read

A runbook step that said “confirm the health endpoint returns 200 after applying” is an assertion, and assertions belong in the tool. A failing check is a warning…

Import blocks, in a plan somebody can review

NoteJuly 19, 20232 min read

Adopting an existing resource used to be a command run against state, invisible to review and impossible to repeat. Seeing the drift in the plan is the improvement…

Two services that share a deploy, which is one service

NoteJuly 17, 20232 min read

Two repositories, two containers, one pipeline that deploys them together and a rollback that has to move both. The distribution is real and the independence is not, which…

The state file two people wrote to at once

Deep diveJuly 14, 202310 min read

Two pipelines, one state file, and a plan that proposed deleting a database. A lock held by a run cancelled forty minutes earlier.

Partitioning by month, and the query that scanned all of them

Deep diveJune 27, 202310 min read

A 400 million row events table and a delete that could not finish. Dropping a partition takes two seconds; getting there takes a schema change.

A cold cache after every deploy, and the warmer we wrote

NoteJune 22, 20231 min read

A deploy that flushes the cache produces two minutes of every request doing the expensive thing at once. A warmer is an admission that the cache is load-bearing,…