DNS failover, and the resolver that ignored the TTL

A sixty-second TTL, and traffic that took forty minutes to move.

the drill, measured by requests per second on each
host after the DNS change:

  0-4 min     88% moved
  4-12 min    97%
  12-40 min   99.4%
  40+ min     a long tail, mostly one corporate
              resolver and a mobile network

the tail is not a bug. a TTL is a request rather than
an instruction, and some resolvers floor it.

which means failover time is 4 minutes for most and
unbounded for a few, and the runbook now says so.

A sixty-second TTL produces four-minute failover for the great majority and a long tail that cannot be controlled, which is the honest characterisation and is worse than the number anybody would have quoted. A load balancer would remove the tail and adds a component whose failure is the thing it was meant to protect against.