A new machine to a running copy of the site in an afternoon. That was the claim at the top of the README, which ran to two pages of numbered steps and had been accurate in September. By March three people had followed it and ended up with three different machines, and the differences only became visible when something broke on one of them. What follows is the replacement: a Vagrantfile describing the machine and an Ansible playbook describing everything inside it, both sitting in the repository next to the code they support.
The symptom
The bug was a timestamp in a CSV export landing an hour out, and it happened on exactly one laptop. Two hours went into the export code before anybody thought to compare the machines rather than the branches.
$ php -v
PHP 5.4.9-4ubuntu2.4 (cli)
$ php -v
PHP 5.5.9-1ubuntu4 (cli)
$ php -v
PHP 5.3.10-1ubuntu3.9 (cli)
$ php -i | grep date.timezone
date.timezone => no value => no valueThree patch versions, and underneath them three php.ini files nobody had ever compared. The machine with no date.timezone was falling back to whatever the operating system believed, which on that one laptop was not what the server believed. The export code was fine. It had always been fine.
That is the class of problem worth fixing, not the individual bug. A defect that reproduces on one machine out of three costs several times what it should, because the first hours go into the wrong question.
Why it happens
The box is built by hand from instructions that were accurate once. Every step in a README is a snapshot of what somebody did on one afternoon, and it starts decaying immediately: a package changes its default, a PPA moves, an extension gets renamed, a step turns out to have depended on something already installed on the author’s machine and never written down.
Nothing executes a README, so nothing tells you it has gone stale. It is discovered stale, by the next person to follow it, usually on the day they are least able to spend the afternoon on it. And because each person patches over their own failure locally rather than fixing the document, the three resulting machines diverge further with every hire.
The fix is not a better document. It is to make the description of the machine executable, so that following it is the only way to build one and any inaccuracy fails loudly on the spot.
There is a second reason to want this, which has nothing to do with new starters. Anything installed by hand cannot be reviewed. A change to a server made over SSH leaves no diff, no author and no reason, and the only record that it happened is that the machine now behaves differently from the one next to it. Moving the description into the repository puts server configuration through the same review as application code, which turns out to be the part that lasts longest.
The fix
The machine
Vagrant describes the virtual machine and nothing about its contents: which base image, how much memory, which private address, which directory is shared with the host. That file is short and it almost never changes, which is exactly the property you want from the outer layer.
# -*- mode: ruby -*-
Vagrant.configure("2") do |config|
config.vm.box = "precise64"
config.vm.box_url = "http://files.vagrantup.com/precise64.box"
config.vm.hostname = "shop.dev"
config.vm.network "private_network", ip: "192.168.56.20"
config.vm.synced_folder ".", "/var/www/shop", nfs: true
config.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--memory", "1024"]
vb.customize ["modifyvm", :id, "--cpus", "2"]
# without this, apt-get resolves through the NAT stack and crawls
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
end
config.vm.provision "ansible" do |ansible|
ansible.playbook = "provisioning/site.yml"
ansible.inventory_path = "provisioning/inventory/development"
end
endThe private network address is worth pinning rather than leaving to DHCP, because it ends up in a hosts file entry and in the Xdebug configuration, and a machine whose address changes on every rebuild makes both of those a recurring chore.
Everything inside it
The playbook is the part that carries the knowledge. A play names the hosts and the roles that apply to them, and that top-level file stays readable precisely because everything hard has been pushed down into a role.
---
- hosts: web
sudo: yes
vars_files:
- vars/development.yml
roles:
- common
- nginx
- php
- mysql
- redisThe PHP role is the one that would have prevented the export bug. It pins the version, installs a named list of extensions rather than whatever the meta-package pulls in this month, and writes the two php.ini values that differ from the package defaults instead of leaving them to chance.
---
- name: php 5.5 from a maintained ppa
apt_repository: repo='ppa:ondrej/php5' update_cache=yes
- name: the extensions the application actually loads
apt: pkg={{ item }} state=present
with_items:
- php5-fpm
- php5-mysqlnd
- php5-curl
- php5-gd
- php5-intl
- php5-mcrypt
- name: the ini values that must not differ between machines
lineinfile:
dest={{ item.file }} regexp='^;?{{ item.key }}' line='{{ item.key }} = {{ item.value }}'
with_items:
- { file: '/etc/php5/fpm/php.ini', key: 'date.timezone', value: 'UTC' }
- { file: '/etc/php5/fpm/php.ini', key: 'memory_limit', value: '256M' }
- { file: '/etc/php5/cli/php.ini', key: 'date.timezone', value: 'UTC' }
notify: restart php5-fpmTwo entries for date.timezone, because the CLI and the FPM pool read different files and the export that started all of this ran from cron. A README would have said “set your timezone”; the playbook has to say where.
Warning
The Ansible provisioner runs Ansible on the host, and Ansible does not run on Windows. The two Windows machines get a shell provisioner instead: install Ansible inside the guest, then run the same playbook against localhost. Same roles, same result, one more line to keep working.
A role per service, so the same playbook builds the server
Splitting by service rather than by machine is what makes this worth the effort. A role knows how to install and configure one thing; a machine is a list of roles plus a set of variables. The development box and the VPS then differ in the variables, not in the steps.
# provisioning/inventory/development
[web]
192.168.56.20 ansible_ssh_user=vagrant
# provisioning/inventory/production
[web]
shop.example.com ansible_ssh_user=deploy
[web:vars]
nginx_worker_processes=2
mysql_buffer_pool=2GThis is the point at which the playbook stops being a development convenience. A configuration change to nginx is now made once, in a role, and reaches both machines — and the box on the desk is genuinely built by the same instructions as the one taking traffic, rather than by an approximation of them.
It also imposes a discipline that is uncomfortable at first: anything done to the server over SSH is lost at the next run, so it has to go into a role instead. That is the whole benefit, arriving as an inconvenience.
Where the data comes from
A machine with no database in it is not a development environment, and this is the step people leave out of the playbook and keep doing by hand. The role that creates the schema also loads a seed file, and the seed file is checked in — small, deliberate and readable, rather than a copy of somebody’s database from last spring.
---
- name: the application database and its user
mysql_db: name={{ app_db }} state=present
- mysql_user:
name={{ app_db_user }} password={{ app_db_password }}
priv={{ app_db }}.*:ALL state=present
- name: has it been seeded already
command: mysql -N -B -e "SELECT COUNT(*) FROM {{ app_db }}.products"
register: seeded
failed_when: false
changed_when: false
- name: load the checked-in seed
mysql_db: name={{ app_db }} state=import target=/var/www/shop/db/seed.sql
when: seeded.rc != 0 or seeded.stdout|int == 0The guard is what makes the role safe to run repeatedly: import when the table is empty or missing, do nothing otherwise. Without it, every provisioning run silently discards whatever the developer was working on, and they learn to stop running it — which returns the machine to being hand-built within a month.
The seed is deliberately not production data. It has one of everything the code has a branch for: a variable product, a product with no image, an order in every status, a customer whose address has no postcode. It is far more useful than volume for the class of bug that development finds, and it fits in a file that can be read in a review.
The values that must not be in the repository
Development credentials are not secrets. The MySQL password on a box reachable only from the host is vagrant, it is checked in, and pretending otherwise buys nothing. Production credentials are a different file with the same shape, and the difference has to be enforced by something other than good intentions.
$ ansible-vault create provisioning/vars/production.yml
Vault password:
$ head -2 provisioning/vars/production.yml
$ANSIBLE_VAULT;1.1;AES256
39633264653164363933623831306566373538383666326433383635...
$ ansible-playbook -i inventory/production site.yml --ask-vault-passEncrypted at rest, in the repository, decrypted only by whoever has the vault password. The file diffs as a wall of hex, which is the one real annoyance, and it is a fair price for having the production variables version-controlled at all rather than living in one person’s home directory.
They get in anyway, of course. Somebody adds a database password to a template while debugging and commits it; somebody pastes a real API key into the development variables because that is the only way to test the integration. The defence that actually works is a pre-commit grep for the shapes of things — long hex strings, BEGIN RSA PRIVATE KEY, anything matching the production hostname — plus the understanding that a key committed once is compromised and must be rotated rather than removed in a later commit.
The application also has to stop caring which machine it is on. Reading configuration from the environment, or letting the application work out its environment from the hostname, both beat a config file with a commented-out block for each server, because the commented-out block is the one that eventually gets uncommented on the wrong machine.
Verifying it worked
The test is destruction. A provisioning setup that has never been run from nothing is a description of the machine somebody already has, and the difference surfaces on the day a new person joins — which is precisely the day it must not.
$ vagrant destroy -f && vagrant up
...
PLAY RECAP ******************************************************
192.168.56.20 ok=64 changed=61 unreachable=0 failed=0
real 19m41s
$ vagrant ssh -c 'cd /var/www/shop && vendor/bin/phpunit'
OK (212 tests, 588 assertions)
$ vagrant ssh -c 'php -i | grep date.timezone'
date.timezone => UTC => UTCTwenty minutes, unattended, on a machine nobody had touched. Running it on a Friday afternoon once a fortnight is enough to keep it honest; the failures are almost always a package that moved or a repository key that expired, and finding those deliberately is much cheaper than finding them during onboarding.
Tip
Run the playbook twice in a row and check the second run reports changed=0. A task that reports a change every time is not idempotent — usually a command that should have been a module — and it will happily do something destructive on the production run.
What this costs
Twenty minutes of provisioning per rebuild, which is real time and is felt most by whoever is trying to fix something urgently. Most of it is apt-get, so a local package cache or a base box with the packages already in it takes it under five — at the cost of a base box that now has to be rebuilt periodically itself.
The larger cost is that the playbook is a second thing to keep current, and it decays exactly like the README did if nobody runs it. The difference is that its decay is detectable: a stale README stays plausible indefinitely, whereas a stale playbook fails on the next full run. That only holds if there is a full run, which is why the scheduled rebuild is not optional.
And the machine is still not production. It has the same packages, the same versions and the same configuration files, which covers the class of bug that started this — but it has a hundred rows where production has four million, no mail, no cron, and a certificate nobody bought. That gap is real and it turns out to be where the next problem lives.