Automatic security updates with automatic reboots, on a host that runs queue workers with no drain step.
// /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";
// what was missing: anything that stops the workers cleanly
// a job mid-flight at 04:00 was killed, retried, and
// duplicated a payout twice in eight months.The reboot is right and the missing piece is a systemd unit ordering that stops workers before shutdown and gives them a timeout long enough to finish the current job. TimeoutStopSec defaults to 90 seconds, which is shorter than the longest job here, so the drain has to be a signal the worker understands rather than a kill. Turning the reboot off would have been the easy answer and the wrong one.