The main application host was provisioned in 2019 and has been upgraded in place three times since. The provisioning script that supposedly describes it has been maintained throughout and has never once been executed against an empty machine, which means it describes the host in the way a map drawn from memory describes a city.
The symptom
$ uptime
up 412 days
$ ls -la /etc/apt/sources.list.d/
-rw-r--r-- ondrej-php.list 2019-08-14
-rw-r--r-- nodesource.list 2020-03-02
-rw-r--r-- docker.list 2021-11-08
-rw-r--r-- some-vendor.list 2022-04-19 ← ?
$ dpkg -l | wc -l
1,204
$ ./bin/provision --list-packages | wc -l
188Twelve hundred packages against a hundred and eighty declared, most of which is dependency closure and some of which is not. The 2022 repository is for a tool nobody could name, still enabled, still being contacted on every apt update.
Why it happens
A host that works is a host nobody rebuilds, and a script that runs only against an already-working machine converges to describing that machine rather than producing it. Every incident adds one line to the host and zero to the script.
The fix
The inventory, before anything else
# packages the host has and the script does not declare
ssh app-1 'dpkg-query -W -f="${Package}n"' | sort > /tmp/actual
./bin/provision --list-packages | sort > /tmp/declared
comm -23 /tmp/actual /tmp/declared > /tmp/undeclared
# systemd units enabled and not in the script
ssh app-1 'systemctl list-unit-files --state=enabled'
| awk '{print $1}' | sort > /tmp/units
# and the full kernel parameter set, which is where the
# genuinely dangerous drift lives
ssh app-1 'sysctl -a' | sort > /tmp/sysctl-oldeleven things installed by hand, all during incidents:
htop, jq, ncdu, sysstat, percona-toolkit,
redis-tools diagnostic. keep,
and declare.
a MySQL client at 5.7 a leftover. remove.
a PHP extension for a feature
removed in 2021 remove.
two vendor repositories one unidentifiable.
remove.
a logrotate rule for a log the
application no longer writes remove.Six of the eleven are diagnostic tools somebody installed at three in the morning, and those are worth declaring rather than removing — an incident is not the time to discover jq is missing. The other five were genuine leftovers and one of them was a client at a version that no longer matched the server.
The two that were load-bearing
# /etc/sysctl.d/99-turkerdev.conf, on the old host only
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
# nothing in git. no comment. no ticket.
# found by diffing sysctl -a between the old host and a
# freshly provisioned one — about forty differences, of
# which thirty-eight were distribution defaults that
# had changed between releases.the symptom, on the new host, under the load test:
connections reset at ~1,200 concurrent
the old host: passed the same test in 2023
and a logrotate configuration for the nginx access log
with `copytruncate`, added in 2020 because a reload was
dropping connections. also undocumented, also
load-bearing.Diffing the full kernel parameter set is the technique, and the signal-to-noise is poor — forty differences, two of which matter. Both are now in the provisioning script with a comment naming the load test that justifies them, which is the documentation that should have existed in 2020.
The service nothing depended on
$ systemctl list-units --type=service --state=running
| awk 'NR>1 {print $1}' | while read -r u; do
pid=$(systemctl show -p MainPID --value "$u")
[ "$pid" = 0 ] && continue
ss -lntp 2>/dev/null | grep -q "pid=$pid"
|| echo "$u: no listening socket"
done
td-thumbnailer.service: no listening socket
$ systemctl show td-thumbnailer -p NRestarts --value
912A service restarted nine hundred times over five years, serving a feature removed in 2021, consuming a gigabyte. Nothing alerts on a process that is working perfectly and is not needed, and the restart loop was invisible because Restart=always with no burst limit reports the unit as active.
Rebuilding rather than upgrading
the first run of the provisioning script against an
empty machine:
attempt 1 failed at step 4. a package renamed in
22.04, two releases ago.
attempt 2 failed at step 11. a directory the script
assumes exists, created by a step that
was deleted in 2021.
attempt 3 failed at step 19. a service enabled
before its configuration was written —
which works on a host where the
configuration is already there.
attempt 4 succeeded. 41 minutes.
and the running host had none of these problems,
because it had never been provisioned by this script.Each failure is a step that only works against a host that has already been through it once, which is the signature of a script maintained by applying it to a live machine. Four attempts is the accumulated cost of six years of not testing it, and the fix is a monthly CI job that provisions a throwaway machine and destroys it.
The things that are not in a package list
a package diff catches what was installed. it does not
catch what was edited, and that is the larger set:
/etc/php/8.4/fpm/pool.d/www.conf
pm.max_children raised from 20 to 32 in 2022,
during a launch, and never reduced. the memory
arithmetic for 32 children on a 4 GB host does
not work.
/etc/mysql/conf.d/turkerdev.cnf
innodb_buffer_pool_size, correct and undeclared.
/etc/nginx/conf.d/upload.conf
client_max_body_size, raised for one import
endpoint in 2020.
/etc/logrotate.d/nginx
copytruncate, added in 2020 because a reload was
dropping connections. load-bearing.
and a file in /root/ containing a one-line script
that three cron entries reference.# the technique: everything under /etc that differs from
# what the package manager shipped
ssh app-1 'dpkg -V 2>/dev/null | awk "$1 ~ /5/ {print $2}"'
> /tmp/modified-conffiles
# plus anything under /etc that no package owns at all
ssh app-1 'find /etc -type f' | while read -r f; do
ssh app-1 "dpkg -S '$f' >/dev/null 2>&1" || echo "$f"
done > /tmp/unowned
# 41 modified, 188 unowned — most of which is generated
# and about a dozen of which is somebody's decisionThe dpkg -V pass finds files a package shipped and somebody edited, and the unowned pass finds files nobody’s package created. Between them they produce a hundred and thirty candidates and about a dozen genuine findings, which is a poor ratio and is the only way to get the dozen.
The pm.max_children value is the one that would have caused an incident on the new host in either direction. Thirty-two children at roughly ninety megabytes each is close to three gigabytes on a machine with four, which had been surviving because the workers were never all busy at once — provisioning a fresh host from the declared value of twenty would have halved the capacity, and copying the undeclared thirty-two would have carried the overcommit forward.
The cutover
1 the new host, provisioned, running the same image
tags, with the database pointed at the same
instance. no traffic.
2 a load test against it directly, by address. this
is where the sysctl finding appeared.
3 DNS TTL reduced to 60 seconds, 24 hours ahead.
4 DNS switched. 90% of traffic moved within four
minutes; the last stragglers took 40 minutes.
5 the old host kept, serving nothing, for a
fortnight.
6 destroyed, on purpose, with a note in the
decision record.
what could not be moved atomically: the cron jobs. both
hosts would have run them. disabled on the old host at
step 3, which means a 24-hour window where a failed
cutover means missed jobs.The cron window is the one genuine gap in this plan and it was accepted with a note rather than solved — the alternative is a locking mechanism across two hosts for a twenty-four hour period, which is more machinery than the risk justifies. Destroying the old host deliberately, rather than leaving it, is what stops this being a rebuild with a permanent fallback nobody removes.
Verifying it worked
# a second rebuild, from scratch, after the first
$ time ./bin/provision --host=app-2 --from-scratch
real 22m14s # was 41m across four attempts
$ ./bin/host-diff app-1 app-2
packages: identical
units: identical
sysctl: identical
# a host that is reproducible, twice
$ k6 run --vus 1500 --duration 5m script.js
http_req_failed 0.00% # the sysctl finding
$ free -m | awk '/^Mem:/{print $3}'
2,104 # was 3,180 — the
# thumbnailer
# the monthly job, since February
runs: 2, failures: 0Provisioning a second host and diffing it against the first is the assertion that the script produces a machine rather than describes one. The memory difference is the deleted service and is the only change anybody would have noticed from outside.
What this costs
A rebuild is now the only supported way to change the host, which means a change that would have been one apt install is a script edit, a CI run and a cutover. That is correct and it is a real increase in the cost of a small change, and the pressure to make one by hand at three in the morning has not gone anywhere.
Drift will also return. The monthly provisioning job proves the script works and proves nothing about whether the running host still matches it — the host diff is the check that would catch that, and it is a manual command rather than a scheduled one. Six years from now this post will be written again unless somebody schedules it.
The twenty-four hour cron window is the accepted risk, and it is accepted on the basis that a cutover has never failed here. That is a sample of one, and the honest position is that this plan has a gap which was cheaper to document than to close.