The application has run on one host since 2019. The documented recovery time of ninety-four seconds is the time to reboot, which assumes the host comes back — and in March a customer contract arrived with an availability commitment that made that assumption unacceptable.
The symptom
the recovery figures, all measured in drills:
a container crash 14s, automatic
a reboot 94s, automatic
the runtime stopping 4m, needs a human
the host failing unbounded. a rebuild
from the provisioning
script plus a restore,
measured at 64 minutes
on a good day.
and the commitment: 99.9% monthly, which is 43 minutes.Sixty-four minutes for a host failure against a monthly budget of forty-three is the arithmetic that made this a project. One host failure a year exceeds the commitment on its own, and the previous decision — documented and reviewed three times — had been correct precisely because no such commitment existed.
Why it happens
A second host is a project and one host has never failed permanently, so the cost is concrete and the risk is hypothetical. That is the correct read until somebody signs a contract, at which point the risk acquires a number.
The fix
What the requirement actually is
the contract says 99.9% monthly, and the useful
questions are the ones it does not answer:
measured how? agreed: successful responses
to a health endpoint, from an
external monitor, one-minute
resolution.
planned maintenance? excluded, with 72 hours'
notice.
what counts as down? the checkout path. a slow
report is not an outage.
the penalty? a service credit. not
termination.
which turns an abstract requirement into a number and
a scope, and the scope is what makes it achievable.Three architectures
active-passive, manual promotion
a standby that replicates and serves nothing.
failover is a human decision plus a runbook.
recovery: ~10 minutes, dominated by the human.
cost: one more host.
active-passive, automatic promotion
the same, with a consensus mechanism deciding.
recovery: ~60 seconds.
cost: a third node to break ties, and a split-brain
risk that is now automated.
active-active
both serving, a load balancer, and a database that
accepts writes in one place regardless.
recovery: seconds, for the application tier only.
cost: session affinity, a load balancer whose own
failure is the thing being protected against, and
a much larger change.
chosen: the first.Automatic promotion was rejected because a consensus mechanism is a component that fails in ways this team has never debugged, and its failure mode is two writable databases. Ten minutes of manual failover against a forty-three minute monthly budget leaves room for four incidents a month, which is enough.
The database
-- on the standby
CHANGE REPLICATION SOURCE TO
SOURCE_HOST='10.0.1.2', SOURCE_USER='repl',
SOURCE_AUTO_POSITION=1;
START REPLICA;
SET GLOBAL read_only = ON;
SET GLOBAL super_read_only = ON;
-- and a heartbeat, written by the primary every second,
-- which is the only honest lag measurement
CREATE TABLE heartbeat (id TINYINT PRIMARY KEY, beat DATETIME(6));# the promotion, and the guard that comes first
lag=$(mysql -h "$PRIMARY" -N -e
'SELECT TIMESTAMPDIFF(SECOND, MAX(beat), NOW()) FROM heartbeat'
2>/dev/null) || lag=unreachable
if [ "$lag" != unreachable ]; then
echo "the primary is REACHABLE and $lag seconds behind."
echo "this is not a failover. investigate."
exit 1
fi
mysql -e 'STOP REPLICA; RESET REPLICA ALL;
SET GLOBAL super_read_only = OFF;
SET GLOBAL read_only = OFF;'Refusing to promote while the old primary answers is what prevents two writable databases, and it means a network partition between the two produces a refusal rather than a split brain. That is the correct failure — a failover that will not proceed is recoverable and two primaries are not.
The application, and the state that could not be replicated
uploads already in an object store since 2023.
no change.
the cache shared since 2020. no change.
sessions in the cache, which everybody assumed
made them shared. they are — and the
session cookie's domain attribute was
set per host in an nginx include that
had been copied.
→ host B issued cookies for a slightly
different domain, so a failover
logged everybody out.
the ACME
account key not in the provisioning script. the
new host would have registered a
second account and hit a rate limit.
a cron entry added in January for a one-off
backfill that finished in February,
still present, and would have run
twice.Three findings, none of them configuration and all of them state that a provisioning script does not describe. The session cookie is the one that would have made a successful failover look like a failure — everybody logged out is indistinguishable from an outage to the person it happens to.
Traffic
DNS with a 60-second TTL, rather than a load balancer.
measured in a drill, by requests per second on each
host after the change:
0-4 min 88% moved
4-12 min 97%
12-40 min 99.4%
40+ min a long tail — one corporate resolver and
a mobile network, both of which floor the
TTL
which means failover is four minutes for most and
unbounded for a few. the runbook says so.
a load balancer removes the tail and introduces a
component whose failure is the thing being protected
against.A TTL is a request rather than an instruction and some resolvers floor it, which is the honest characterisation and is worse than the number anybody would have quoted. Four minutes for the great majority is within the budget; the tail affects a small number of clients for an unbounded period and there is no way to fix it with DNS.
The failover drill
first drill, announced, on a Saturday:
00:00 the primary is stopped
00:02 the alert fires
00:04 a human acknowledges
00:06 the promotion guard runs and passes
01:10 the database is promoted
02:40 the application configuration is updated —
a file edit, because the write host is a
constant baked into the image
04:10 DNS changed
08:20 88% of traffic on the new host
09:40 the cache is warm
9 minutes 40 seconds, against a runbook estimate of
four. and four steps the runbook did not have.the four missing steps:
1 the write host is a constant in the image. a
promotion needs a deploy, or an environment
variable — we changed it to the latter.
2 the monitoring exporter connects with a user that
has REPLICATION CLIENT on the old primary only.
3 the cache is cold on the new host, and the warmer
runs on deploy rather than on promotion.
4 WP-CLI needs a working database connection, so
step 1 has to be a file edit rather than an
option update.
second drill, unannounced, in April: 6 minutes 10.A runbook that has never been executed is a hypothesis, and four missing steps out of eleven is the normal rate. The unannounced second drill is the one that matters — the first one was performed by the person who wrote the runbook, which tests the runbook against the knowledge that produced it.
Verifying it worked
$ ./bin/host-diff app-1 app-2
packages: identical units: identical sysctl: identical
$ ./bin/replication-lag --p99 --since=30d
180ms
$ ./bin/failover-drill --unannounced
time to serve from the standby: 6m 10s
requests failed during the window: ~1,100
$ ./bin/availability --since=2026-03 --external-monitor
99.97%
# and the guard, tested deliberately
$ ./bin/promote --dry-run # with the primary running
the primary is REACHABLE and 0 seconds behind.
this is not a failover. investigate.Six minutes ten against a forty-three minute monthly budget leaves room for six failovers a month, which is comfortable. The eleven hundred failed requests during the window are the honest cost — a manual failover is not seamless and the commitment is about availability rather than about perfection.
What this costs
Two of everything: two hosts to patch, two to monitor, two provisioning runs to keep identical, and a host diff that is a manual command rather than a scheduled one. The standby serves nothing and costs the same as the primary, which is the price of the commitment and is the cheapest of the three architectures.
A split-brain risk that is now real rather than theoretical. The guard refuses to promote while the primary answers, which is correct and means a partition produces a refusal — and a refusal during a genuine outage is an extra decision for a human at three in the morning, which is exactly when nobody should be making one.
And the decision has to be made by a person, which is the whole design and is the part that will fail. Six minutes assumes somebody is awake, acknowledges within two, and follows eleven steps correctly. The published commitment is business hours with best effort overnight, which three people can honestly offer and which the contract does not quite say.