PHPUnit 10, and the listeners that no longer exist

PHPUnit 10 shipped in February. The headline is a speed improvement that turns out to be modest on a suite like ours, and the actual work is a rewritten extension API, attributes replacing annotations, and a configuration file that had been accreting since 2014.

The symptom

$ vendor/bin/phpunit 2>&1 | head -20
PHPUnit 9.6.4

Warning: Your XML configuration validates against a
deprecated schema.
Suggestion: Migrate your XML configuration using
"--migrate-configuration"!

Deprecation: The "verbose" attribute is deprecated
Deprecation: TestListener is deprecated
Deprecation: Metadata in doc-comments is deprecated
...

41 deprecations, and a custom TestListener that 10.0
removes entirely.

The listener is the blocker rather than the deprecations. It measures slow tests and counts database queries per test, both of which had become part of how the team works, and neither of which has a drop-in equivalent.

Why it happens

The extension API was a nine-method interface where most implementations left seven methods empty, and it exposed internals that constrained every refactor of the runner. Replacing it was overdue and it is a genuine break rather than a rename.

The fix

The configuration, migrated and then read

$ vendor/bin/phpunit --migrate-configuration
Created backup:         phpunit.xml.bak
Migrated configuration: phpunit.xml

$ git diff phpunit.xml | head -20
-  verbose="true"
-  convertDeprecationsToExceptions="true"
-  printerClass="AppTestsProgressPrinter"
-  <filter><whitelist><directory>src</directory>
+  <source><include><directory>src</directory>

The printerClass removal is the interesting line: a custom output formatter written in 2016, maintained through three major versions, silently dropped. Nobody had run the suite with it in months because CI uses the JUnit output, which is how a thing survives that long without being noticed.

The event system

use PHPUnitEventTestFinished;
use PHPUnitEventTestFinishedSubscriber;

final class SlowTestSubscriber implements FinishedSubscriber
{
    /** @var array<string, float> */
    private array $slow = [];

    public function notify(Finished $event): void
    {
        $ms = $event->telemetryInfo()->durationSincePrevious()->asFloat() * 1000;

        if ($ms > 500.0) {
            $this->slow[$event->test()->id()] = $ms;
        }
    }
}

final class TurkerDevExtension implements Extension
{
    public function bootstrap(
        Configuration $configuration,
        Facade $facade,
        ParameterCollection $parameters,
    ): void {
        $facade->registerSubscriber(new SlowTestSubscriber());
    }
}

// phpunit.xml
// <extensions>
//   <bootstrap class="AppTestsTurkerDevExtension"/>
// </extensions>

Thirty lines became fifteen and the typed event is genuinely nicer to work with than a method receiving a Test and a float with no context. The subscriber interface is one method, and registering several is the normal case rather than an awkward one.

The listener that had no equivalent

// the query counter had hooked startTest/endTest to reset
// and read a connection-level counter. there is no event
// carrying the application's state, and there should not
// be — so it moved to where it belongs:

trait CountsQueries
{
    protected function setUp(): void
    {
        parent::setUp();
        DB::enableQueryLog();
    }

    protected function assertQueryCountBelow(int $max): void
    {
        self::assertLessThan($max, count(DB::getQueryLog()));
    }
}

Moving it into a trait on the test case is better than it was: the count is asserted by the tests that care rather than reported for all 1,400, and the reporting version had been producing a number nobody acted on. The migration forced a question that should have been asked in 2019.

Annotations to attributes, across 1,400 tests

$ vendor/bin/rector process tests 
    --config=rector-phpunit10.php --dry-run
  312 files would be changed

$ vendor/bin/rector process tests --config=rector-phpunit10.php
$ git diff --stat | tail -1
 312 files changed, 1,204 insertions(+), 1,388 deletions(-)

# and the two it could not resolve:
#   @dataprovider  (lowercase p) — never ran, 2 tests
#   @expectedException — removed in PHPUnit 9, still
#     present in a file excluded from the suite

The lowercase @dataprovider is the reason attributes are an improvement rather than a change: a misspelled annotation is silently ignored and a misspelled attribute is a class-not-found error. Those two tests had been running once with no data since 2020 and passing.

Static data providers

// what eleven of ours looked like
public function validOrders(): array
{
    return [
        'simple'   => [$this->makeOrder(lines: 1)],
        'discount' => [$this->makeOrder(discount: 10)],
    ];
}

// what they had to become
public static function validOrders(): array
{
    return ['simple' => ['simple'], 'discount' => ['discount']];
}

#[DataProvider('validOrders')]
public function testItAccepts(string $case): void
{
    $order = $this->makeOrder(...self::CASES[$case]);
    // ...
}

The restriction is correct — providers ran before the test case was set up and depending on instance state worked by accident. Passing a key and building inside the test is more code and makes the setup visible, which the previous version hid behind a provider that looked declarative.

The deprecation triggerer, which is new and useful

<phpunit
    failOnDeprecation="true"
    failOnNotice="true"
    failOnWarning="true"
    displayDetailsOnTestsThatTriggerDeprecations="true">
  <source ignoreSuppressionOfDeprecations="false">
    <include><directory>src</directory></include>
  </source>
</phpunit>

what this reports that 9.6 did not: WHICH of our files
triggered a deprecation in a dependency.

  1 test triggered 4 deprecations:
    src/Reporting/Exporter.php:88
      Passing null to parameter #1 of str_replace()
      is deprecated

the previous version reported the deprecation and the
test. this reports the line in OUR code that caused it,
which is the difference between a list and a task.

Failing the build on deprecations is only tolerable because the report names the calling line — a list of deprecations from inside a dependency is unactionable, and a list of our own lines is a morning’s work. Turning it on found nine, of which seven were the same null-to-non-nullable pattern from a single helper.

The output format, and the CI parser that broke

the pipeline had a step parsing the text output for a
count of skipped tests. the format changed.

what it should have been using all along:
  --log-junit    a schema, stable across versions
  --log-teamcity for the runner that understands it

parsing human-readable output is a dependency on a thing
nobody promised to keep stable, and it broke on a minor
version twice before it broke on this major.

The speed claim, measured

# same machine, same tests, cold
$ time vendor/bin/phpunit      # 9.6
real    14m12s

$ time vendor/bin/phpunit      # 10.0
real    12m54s                 # -9%

# where it came from
  metadata parsing + discovery   70s → 8s
  everything else                unchanged

# 1,100 of our tests open a transaction against MySQL.
# the framework was never the bottleneck.

Nine per cent is worth having and is not what the announcement suggests, because the announcement is measured on a suite of fast unit tests where framework overhead dominates. Measuring before upgrading costs one command and makes the justification honest — the reason to do this upgrade is the deprecations and the attributes, not the speed.

Verifying it worked

$ vendor/bin/phpunit
PHPUnit 10.0.16
Tests: 1,414, Assertions: 4,208, Skipped: 2
# 1,414 not 1,412 — the two @dataprovider tests now run

$ vendor/bin/phpunit --coverage-text | tail -3
  Lines:   74.12% (8,412/11,348)
# identical to 9.6, to two decimal places

$ vendor/bin/phpunit 2>&1 | grep -ci deprecat
0

Coverage being identical is the assertion that the migration changed nothing about what runs. The test count going up by two is the one difference, and it is the bug the upgrade found rather than a regression — both of those tests pass, which is lucky.

What this costs

A major version bump that touched 312 test files, most of it mechanical and all of it in the diff of a single pull request that nobody could review line by line. The review was of the Rector configuration and a sample, which is the only practical approach and is a weaker guarantee than usual.

The extension is also a thing we now maintain against an API that is new enough to still be moving. The previous listener survived three major versions largely unchanged; there is no reason to expect the same of this one, and the query-counting trait — which depends on nothing PHPUnit provides — will outlast it.