A library that claims 5.3 compatibility is tested against whatever the author has installed, which is one version. The failures that reach users are the ones that version does not have — a short array literal in a 5.3 codebase, a finally block, an ext/mysql call that a newer build no longer ships. A build matrix is a config file that runs the suite against each of them.
language: php
php:
- 5.3
- 5.4
- 5.5
- 5.6
- hhvm
matrix:
fast_finish: true
allow_failures:
- php: hhvm
before_script:
- composer install --no-interaction --prefer-source
script: phpunit --configuration phpunit.xml.distFive jobs run in parallel and the build is red if any of them is, which is the point — but allow_failures is what makes a matrix usable rather than infuriating. HHVM is worth having in the list this year and not worth blocking a merge on. fast_finish reports the result as soon as every non-excused job has finished, so a slow HHVM run does not hold up the answer. The costs are honest: five times the build minutes, and a composer.json whose dev dependencies have to install cleanly on 5.3, which usually means pinning PHPUnit lower than you would like. A second axis multiplies rather than adds — five PHP versions against two database versions is ten jobs — so add one only for the thing that genuinely differs.