A WordPress plugin with a test suite that runs in CI

The plugin ran on forty client sites across three WordPress versions and two PHP versions, and the release process was a person opening six of them and clicking around. A 5.7 upgrade broke it on the sites still running PHP 7.4, which nobody discovered for nine days.

The symptom

$ ls tests/
ls: cannot access 'tests/': No such file or directory

$ cat .github/workflows/*.yml 2>/dev/null | head
# nothing

# and the release checklist, in a wiki page:
#   1. bump the version
#   2. check on staging
#   3. check on the three biggest client sites
#   4. deploy to the rest
#
# step 3 was the test suite, and it covered three
# configurations out of six.

Three of six configurations, checked by hand, by whoever was releasing. The failure that reached production was on a combination nobody in the team ran locally.

Why it happens

WordPress is not a framework you instantiate — it is a large amount of global state that must be loaded before any of your code means anything. A unit test for a function calling get_option needs a database, an installed WordPress and a bootstrap, which is enough friction that most plugins have no tests at all.

The fix

The bootstrap, which is the whole barrier

// tests/bootstrap.php
$core = getenv('WP_TESTS_DIR') ?: '/tmp/wordpress-tests-lib';

require_once $core . '/includes/functions.php';

// load the plugin at the right moment — muplugins_loaded,
// before WordPress decides what is active
tests_add_filter('muplugins_loaded', static function (): void {
    require dirname(__DIR__) . '/turkerdev-orders.php';
});

require $core . '/includes/bootstrap.php';

The muplugins_loaded filter is the piece that is not obvious: requiring the plugin file directly at the top of the bootstrap loads it before WordPress exists, and requiring it afterwards means the activation hooks and init callbacks have already fired. This is the line that most first attempts get wrong.

The test library is a separate checkout from WordPress itself, and the install script that fetches both is the other half. Running it in CI rather than expecting a developer to have run it once is what makes the suite reproducible.

The matrix, which is the actual value

jobs:
  test:
    runs-on: ubuntu-20.04
    strategy:
      fail-fast: false
      matrix:
        php: ['7.4', '8.0']
        wordpress: ['5.6', '5.7', 'trunk']
        exclude:
          - { php: '8.0', wordpress: '5.6' }   # 8.0 lands in 5.6.1
    services:
      mysql:
        image: mysql:8.0
        env: { MYSQL_ROOT_PASSWORD: root }
        ports: ['3306:3306']
        options: --health-cmd="mysqladmin ping" --health-interval=5s

Five cells, run on every push, is the thing that could not be done by hand — and it is the only part of this that would have caught the actual incident. fail-fast: false is what makes the matrix informative rather than reporting the first failure and cancelling the rest.

Including trunk is the early warning system: it fails when WordPress changes something before the release, which is a message rather than a build failure and belongs on a non-blocking job.

    steps:
      - uses: actions/checkout@v2
      - uses: shivammathur/setup-php@v2
        with:
          php-version: ${{ matrix.php }}
          extensions: mysqli, zip
          coverage: none
      - run: bash bin/install-wp-tests.sh wordpress_test root root 
               127.0.0.1 ${{ matrix.wordpress }}
      - run: composer install --no-progress
      - run: vendor/bin/phpunit

Testing a hook without a browser

final class OrderStatusTest extends WP_UnitTestCase
{
    public function test_status_change_schedules_a_notification(): void
    {
        $id = $this->factory->post->create(['post_type' => 'shop_order']);

        turkerdev_set_order_status($id, 'shipped');

        $this->assertNotFalse(
            wp_next_scheduled('turkerdev_notify_shipped', [$id])
        );
    }
}

WP_UnitTestCase wraps each test in a transaction and rolls it back, which is what makes a suite against a real database fast enough to run on every push. The post factory is the part that makes fixtures bearable — building a post by hand through wp_insert_post is six lines and a set of defaults nobody remembers.

public function test_the_price_filter_is_applied(): void
{
    $called = 0;

    add_filter('turkerdev_order_total', function (int $cents) use (&$called): int {
        $called++;

        return $cents + 500;
    });

    $this->assertSame(5400, turkerdev_order_total($this->orderId()));
    $this->assertSame(1, $called);
}

Asserting that a filter is applied, and applied exactly once, is the test that protects the plugin’s public interface — a refactor that stops calling apply_filters breaks every site that hooked it, silently. Those are the tests worth writing first on a plugin distributed to other people.

Catching a deprecation before WordPress does

// WP_UnitTestCase FAILS a test that triggers a deprecation
// or a doing_it_wrong. so a deliberate one is declared:
public function test_legacy_shortcode_is_deprecated(): void
{
    $this->setExpectedDeprecated('turkerdev_old_shortcode');

    turkerdev_old_shortcode([]);
}

// and against trunk, an UNdeclared one fails the build —
// weeks before the release ships.

This is the feature that makes the trunk job worth running: WordPress deprecates a function, the suite fails on trunk, and there are two months to fix it before any client site is affected. Without it the sequence is a release, a broken site, and a phone call.

Verifying it worked

$ vendor/bin/phpunit
Tests: 84, Assertions: 211, Time: 00:11.402

# the matrix, in CI
#   7.4/5.6 ✓   7.4/5.7 ✓   7.4/trunk ✓   8.0/5.7 ✓
#   8.0/trunk ✗  1 deprecation   ← the point

$ vendor/bin/phpcs --standard=WordPress src/
0 errors, 0 warnings

The failing trunk cell on the first run is the outcome that justified the whole exercise: a function deprecated in the upcoming release, caught two months early, on a combination nobody would have tested by hand.

The coding standard check is a separate job and is worth having on a plugin that other people read — it is the difference between a codebase that looks like WordPress and one that looks like whoever wrote it most recently.

What this costs

A pipeline that takes eight minutes across five cells and a fixture database created five times, which is the price of the coverage and is unavoidable with this test library. The suite is also slower than a plugin author expects, because every test hits MySQL — there is no meaningful unit-test tier without abstracting WordPress behind interfaces, which is a much larger design decision.

The install script is the fragile part: it downloads WordPress and the test library from a URL scheme that has changed twice, and it will break at some point for reasons unrelated to the plugin. Pinning the versions and vendoring the script rather than curling it from a gist is the difference between a five-minute fix and an afternoon.