Türker Yıldırım

vps

  • Eleven things installed by hand during incidents

    Note2 min read

    A diff between what the provisioning script installs and what is actually on the host. Six of the eleven are diagnostic tools somebody installed at three in the…

  • A host rebuilt from the provisioning script, for the first time

    Note2 min read

    A provisioning script maintained since 2019 and never once run against an empty machine. A script that only ever runs against an already-working host converges to describing that…

  • An on-call rota of three

    Deep dive10 min read

    A rota of two is not a rota. A third person joined in November, and the gap was everything learned in the shadowing month and never written down.

  • Ubuntu 24.04, and the two packages that moved

    Note2 min read

    A rebuild onto the new long-term-support release, which was uneventful except for two packages that had been renamed. The two package errors stopped the build, which is the…

  • Handing over on-call to one other person

    Deep dive10 min read

    One person who can fix anything, and a rota of one. Three years of incidents resolved by the same person, every time.

  • The registry we ran ourselves for three weeks

    Deep dive10 min read

    A rate limit hit at 09:00 every weekday, a pipeline failing intermittently for two months, and the wrong one of three options tried first.

  • Offsite copies, and the egress bill

    Note2 min read

    A second copy in a second provider, and a bill that arrived a month later showing the transfer cost more than the storage. Egress pricing makes a naive…

  • innodb_buffer_pool_size on a machine with other tenants

    Note2 min read

    The advice to set the buffer pool to 70-80% of RAM assumes a dedicated database server, and this one also runs PHP and Redis. The hit rate is…

  • Import blocks, and the resources we had been pretending to manage

    Deep dive10 min read

    Terraform 1.5 in June, and an import that is part of the plan rather than a side effect. Fourteen resources created by hand in 2019.

  • Debian 12 on a host that had been on 10

    Note2 min read

    Two major versions in one jump, which the documentation says not to do and which is what the situation actually was. A rebuild is the honest test of…