The proposal was fourteen pages and had a control plane in it. The question it answered was how to deploy four PHP services onto three machines, which is a question with several answers, and the proposal had arrived before anyone had written down what the requirements actually were. This is what happened when we did.
The symptom
the proposal, abridged:
3 control plane nodes (HA etcd, or it is a single point of failure)
3 worker nodes
an ingress controller
cert-manager
a CNI plugin, chosen from six
persistent volume provisioning
a monitoring stack for the cluster itself
a log aggregation stack for the cluster itself
team size: 4 engineers, none of them full-time on infrastructure
current deployment: rsync and a symlink, which worksEvery line of that is genuinely required for a production cluster, and none of it is the application. The honest reading is that the proposal introduced eight new things that can break in order to solve a problem that had not been stated.
Why it happens
The tooling everyone writes about is built for a scale nobody in the room has. Kubernetes solves the problem of scheduling thousands of containers across hundreds of machines with dozens of teams deploying independently, and it solves it well — the design is not the issue.
The issue is that the material available to somebody researching deployment in 2019 is overwhelmingly about that scale, because that is where the interesting engineering is. There is very little written about deploying four services onto three machines, which is where most people are.
The fix
Naming what you actually need
what we need why
----------------------------------------------------------------
restart a crashed process it happens weekly
roll out without dropping requests deploys are during the day
roll back in under a minute the reason deploys are scary
secrets not in the repository audit finding, has a date
run 2+ copies of the API one machine reboots monthly
know when something is down currently: a customer tells us
what we do NOT need
----------------------------------------------------------------
autoscaling traffic is flat and predictable
multi-tenancy one team, one application
bin packing three machines, six services
self-healing across nodes a node failure is a person's
afternoon, and that is acceptableSix requirements, and the last column is what makes the list honest — each one has a reason that somebody outside engineering would recognise. Writing the second list is the part that changes the conversation: autoscaling and bin packing are most of what a scheduler is for, and neither was wanted.
What one host with systemd gives you
# /etc/systemd/system/api.service
[Unit]
Description=API
After=network.target
[Service]
Type=notify
User=app
ExecStart=/usr/bin/php-fpm7.3 --nodaemonize --fpm-config /etc/app/fpm.conf
ExecReload=/bin/kill -USR2 $MAINPID
Restart=always
RestartSec=2
MemoryMax=1G
EnvironmentFile=/etc/app/secrets.env
[Install]
WantedBy=multi-user.targetThat covers restart-on-crash, a memory limit, a graceful reload and secrets outside the repository — four of the six requirements, in twelve lines, using software already on the machine. EnvironmentFile with mode 600 owned by root is not a secrets manager and it is not the repository either, which is what the audit finding actually said.
What it does not give you is anything across machines. Systemd knows about one host, so running two copies of the API on two machines means two unit files and something in front deciding where requests go — which is the boundary at which one host stops being enough.
What swarm gives you, and where it stops
version: '3.7'
services:
api:
image: registry.internal/api:${TAG}
deploy:
replicas: 4
update_config:
parallelism: 1
delay: 10s
order: start-first
failure_action: rollback
restart_policy: { condition: on-failure }
secrets: [db_password]
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost/ready"]
interval: 10s
start_period: 30s
secrets:
db_password:
external: trueThat covers all six requirements including the two systemd could not, on three machines, with one command to deploy and one to roll back. Secrets are mounted from a tmpfs rather than passed as environment variables, which is a genuine improvement over the systemd version.
Where it stops is worth being specific about. Swarm has no autoscaling, a much smaller ecosystem, storage orchestration that is barely there, and — the part that matters most in 2019 — an uncertain future, since Docker’s own attention had visibly moved. Choosing it means choosing something that may not be maintained in three years.
$ docker stack deploy -c stack.yml --with-registry-auth app
$ docker service ps app_api --format '{{.Name}} {{.CurrentState}}'
app_api.1 Running 4 minutes ago
app_api.2 Running 4 minutes ago
$ docker service rollback app_api
app_api: rolling back to previous specification
verify: Service convergedWhat it would have cost to be wrong in the other direction
Arguing against a scheduler is only honest if the cost of not having one is stated, so it is worth writing down what the chosen arrangement genuinely cannot do.
what we gave up, and what it costs when it bites
a node dies at 02:00 someone gets up. 3 replicas on 2 nodes
keeps serving; capacity is degraded
until morning. acceptable, stated.
a service needs 8 copies the machines cannot hold them.
adding a node is a provisioning
ticket and an afternoon, not an API call.
a rollback across services each stack is deployed separately,
so a coordinated rollback is four
commands and a runbook.
storage for a stateful swarm's volume support is thin.
service anything with state stays on a host,
deliberately, forever.The first row is the one worth agreeing out loud with whoever owns the service level, because it is the only one that affects customers and it is the one that gets glossed over. “Degraded capacity until somebody wakes up” is a policy, and it is a defensible policy for a business whose overnight traffic is a tenth of daytime — which was the actual situation and had never been checked.
The last row is the one that should not be argued about at all. A database in a scheduler is a genuinely hard problem that the operators exist to solve and that a small team should not be solving; keeping stateful services on named hosts with ordinary backups is the right answer regardless of what runs the stateless ones.
The three questions that decide it
how many hosts?
1 systemd. anything else is overhead.
2-5 swarm, or systemd plus a load balancer and a script.
6-20 the argument starts being real.
20+ a scheduler. the question is which.
how many people deploy?
1 team any of the above.
3+ teams you need isolation, quotas and RBAC. that is Kubernetes.
how often does the shape change?
rarely a static topology is fine, and cheaper.
weekly declarative infrastructure starts paying for itself.Three hosts, one team, a topology that had not changed in two years. Every axis pointed the same way, which is not always the case — a team of three with twelve services changing weekly is a genuinely harder call and the answer might be different.
The number that is usually missing from this conversation is the operational cost. A cluster needs upgrading roughly quarterly, and a Kubernetes upgrade is a scheduled piece of work with a rollback plan, not a package update. On a four-person team with no dedicated infrastructure engineer, that is a meaningful fraction of somebody’s year spent on the platform rather than on the product.
When the answer flips, and what to have built by then
Deciding against something is only defensible if the conditions for changing the decision are written down, and if the work done in the meantime does not have to be thrown away.
revisit when any of these is true:
more than one team deploys independently
more than about eight services
traffic varies enough that fixed capacity is wasteful
somebody is employed to run infrastructure
what transfers regardless of the answer:
containers, and a Dockerfile per service ✓ done
a health endpoint that checks dependencies ✓ done
configuration and secrets from the environment ✓ done
stateless application processes ✓ done
a registry, with immutable tags ✓ done
structured logs to stdout ✓ doneEverything in the second list is required by any scheduler and is worth doing on one host regardless, which is what makes the decision reversible. A team that has done all six has already paid most of the cost of a migration; the remaining work is the manifests, which is days rather than months.
That is the argument worth making to whoever wanted the fourteen-page proposal: not “no”, but “not yet, and here is the work that makes yes cheap when it arrives”. It is a considerably easier conversation than a refusal.
Verifying it worked
# a rolling deploy, with traffic
$ while true; do curl -s -o /dev/null -w '%{http_code}' https://api.internal/ready; done &
$ docker stack deploy -c stack.yml app
200200200200200200200200200200200200 # no 502s, no gaps
# a node failure, caused on purpose, in the afternoon
$ docker node update --availability drain node-02
$ docker service ps app_api --filter desired-state=running | wc -l
5 # 4 replicas, redistributed
# and the rollback
$ time docker service rollback app_api
real 0m22.418sDraining a node deliberately, during working hours, is the test that says the arrangement meets the requirement — and it is the one people skip because it feels risky. Doing it on a Wednesday afternoon with everybody watching is how you find out whether the health checks are meaningful, and the answer on the first attempt was that they were not: the check returned 200 before the application could serve a request.
Twenty-two seconds to roll back is the number that changed how the team felt about deploying, and it is the requirement that had been driving the whole conversation without being written down.
What this costs
A decision that is unfashionable and has to be defended repeatedly, to new joiners, to contractors, and in interviews. “We are not using Kubernetes” reads as a gap on a job advertisement and as a red flag to somebody who has only worked at that scale, and the reasoning has to be re-explained rather than assumed. Writing it down once — the requirements, the two lists, the revisit conditions — is what makes that a document rather than an argument.
The honest risk is that the revisit conditions are never checked. A decision made against a set of assumptions has to be re-examined when the assumptions change, and nothing prompts that automatically — the team grows to eight, the services grow to twelve, and the swarm setup persists because it works and nobody scheduled the review. Putting a date on it rather than a condition is cruder and considerably more likely to happen.