Draining a node on a Wednesday afternoon, on purpose

A failover that has never been exercised is a procedure nobody has tested, and the first execution is always during an incident with an audience.

# announced, in working hours, with everybody watching
$ docker node update --availability drain node-02
$ watch docker service ps app_api

# what it found the first time:
#   the health check returned 200 before the app could serve
#   one service had a bind mount only node-02 had
#   the runbook named a host that was decommissioned in March

Every one of those findings is something that would have been discovered at 03:00 otherwise, and each took ten minutes to fix on a Wednesday. Announcing it beforehand is what makes it a drill rather than an incident. Doing it quarterly is enough, and the value is almost entirely in the findings rather than in the confidence — a drill that finds nothing is a drill that was not ambitious enough.