Moving a pipeline to a runner you do not own

The build worked on one machine. It had worked on that machine since 2017, and every attempt to reproduce it elsewhere had failed for a different reason — a globally installed package, a PHP extension somebody added by hand, a directory created during an incident and never documented. The build was not a script; it was a machine with a script on it.

The symptom

# the pipeline, in its entirety
cd /var/lib/jenkins/workspace/shop
composer install && npm install
./vendor/bin/phpunit && ./vendor/bin/phpcs

# and what it silently depended on
$ composer --version
Composer 1.8.4              # installed 2018, never updated
$ php -m | wc -l
47                          # eleven not in any Dockerfile
$ ls ~/.ssh/
id_rsa                      # a deploy key nobody could account for

Five lines of build script and forty-seven extensions of implicit dependency. The deploy key was the part that ended the discussion: nobody knew when it had been added, what it had access to, or which of the six people with Jenkins access could use it.

Why it happens

A long-lived runner accumulates state, and every accumulation is somebody solving a real problem at the time. A missing extension is installed by hand because the build is red and the release is today; a credential is dropped into the home directory because the alternative is an afternoon of plumbing. Neither is written down because neither felt like a change.

The result is a machine that nobody can rebuild, which means the pipeline is a single point of failure with no recovery procedure — and the failure mode is not that it breaks, it is that it cannot be moved.

The fix

A workflow that assumes nothing

name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2

      - uses: shivammathur/setup-php@v2
        with:
          php-version: '7.4'
          extensions: pdo_mysql, redis, gd, intl, zip
          coverage: none

      - run: composer install --no-interaction --prefer-dist
      - run: vendor/bin/phpunit
      - run: vendor/bin/phpcs

The extension list is the artefact that matters most: forty-seven extensions became five, because listing them forced somebody to establish which were actually used. That list is now in version control and is the same list the Dockerfile uses, which is the first time the two have agreed.

Pinning the runner image rather than using ubuntu-latest is the other habit: latest moves, and a build that broke because the image changed underneath is a bad morning with no commit to blame. coverage: none is worth setting explicitly because the default installs Xdebug, which roughly triples the runtime of every test job for a report nobody asked for.

The cache, which is what makes it bearable

- id: composer-cache
  run: echo "::set-output name=dir::$(composer config cache-files-dir)"

- uses: actions/cache@v2
  with:
    path: ${{ steps.composer-cache.outputs.dir }}
    key: composer-${{ hashFiles('**/composer.lock') }}
    restore-keys: composer-

- uses: actions/cache@v2
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('**/package-lock.json') }}

Caching the download directory rather than vendor/ is the important choice: a stale vendor/ silently ships the wrong versions, and a stale download cache costs nothing because Composer still resolves from the lock file. restore-keys gives a partial hit when the lock changes, so a one-package update does not start from nothing.

# cold 2m41s   warm 1m04s   the old Jenkins job 1m12s
#
# and the number that mattered more — reproducing the build
# on a new machine: never, before. every run, after.

Service containers, and the readiness nothing does for you

Integration tests need a real database, and a hosted runner is an empty machine — which is the problem service containers exist for and the source of the first intermittent failure.

services:
  mysql:
    image: mysql:8.0
    ports: ['3306:3306']
    options: >-
      --health-cmd="mysqladmin ping" --health-interval=10s
      --health-timeout=5s --health-retries=5

# the connection is 127.0.0.1, not a service name — the port is
# published to the HOST, unlike in a compose file.

The health options are not optional: without them the job starts before MySQL is listening, and the failure is an intermittent connection refused that only happens on a cold runner. That intermittency is the worst possible symptom because it looks like flakiness in the tests rather than a race in the pipeline.

Connecting to 127.0.0.1 rather than to a hostname is the difference from a compose file and it catches everyone once. The steps run on the host and the services are containers with published ports, which is the opposite of the local arrangement.

A matrix, and testing two PHP versions for the same effort

strategy:
  fail-fast: false
  matrix:
    php: ['7.3', '7.4']

steps:
  - uses: shivammathur/setup-php@v2
    with: { php-version: "${{ matrix.php }}", coverage: none }

fail-fast: false is the setting worth changing from the default: without it one failing combination cancels the rest and you learn about one problem instead of two. Each combination is a separate runner, so this doubles the minutes — which is worth it for a library and is worth questioning for an application that deploys to one version.

Secrets, and the fork that must not read them

on: pull_request         a fork's PR runs with NO secrets.
                         a workflow needing them fails, correctly.

on: pull_request_target  runs in the BASE repo's context and DOES
                         have them. combined with a checkout of the
                         fork's head, that hands every secret to
                         anybody who can open a pull request.

The absence of secrets on fork pull requests is a feature and it means a test job needing a third-party API key cannot run on them — which is a design constraint rather than a problem to work around. Stubbing the dependency in CI is the answer, and it improves the tests anyway.

For the deploy credential, the mitigation is scope: a token permitted to push one image to one repository is a different risk from the Jenkins deploy key, which had shell access to production. Enumerating what each secret can do was a side effect of the migration and was the most valuable part of it.

Verifying it worked

# the assertion the whole exercise was for
$ gh workflow run ci.yml --ref main
$ gh run watch
✓ test (7.3)   1m 04s
✓ test (7.4)   1m 06s

# from an empty cache, which is the real test
$ gh cache delete --all
$ gh workflow run ci.yml --ref main
✓ test (7.4)   2m 41s

# and the machine that no longer matters
$ ssh jenkins.internal 'sudo systemctl stop jenkins'
# nothing broke.

Deleting the cache and running again is the check that the workflow genuinely assumes nothing, and it is the one people skip because the warm run already passed. Stopping the old server for a week before decommissioning it is the other half — a pipeline nobody has needed for a week is a pipeline that can be deleted.

What this costs

A pipeline that runs on somebody else’s infrastructure, with somebody else’s outage schedule and somebody else’s billing. A CI provider being down means no deploys, and the recovery is waiting — which is a genuine loss of control compared with a machine you can restart. It is worth being honest that this is a trade rather than an improvement in every direction.

The second cost is that the private network is unreachable. A hosted runner cannot see the database, the internal registry or the deploy target, which means either a self-hosted runner inside the network — with its own considerable set of questions about what a fork’s code can reach — or a deploy that is triggered rather than performed by CI. That decision was deferred during this migration and had to be made properly a few months later.