PHPUnit 9 to Pest, and the migration that was not worth it

Pest had been suggested three times in eighteen months and had never been evaluated, which is how a tool becomes a recurring agenda item rather than a decision. In September it was trialled properly on one suite of a hundred and forty tests, with a written rubric agreed beforehand, and the outcome was not to adopt it.

The starting position

the rubric, agreed before the trial:

  1  does it reduce the cost of WRITING a test?
  2  does it reduce the cost of READING a test six months
     later?
  3  does it work with everything we already have —
     coverage, static analysis, the IDE, parallel running,
     the CI reporter?
  4  what is the cost of a half-finished migration?

and the thing that was NOT on the list, deliberately:
  "is it nicer". it is, and that is not a reason.

Agreeing the criteria before looking at the tool is what makes a trial a decision rather than a preference, and the fourth question is the one that matters most for something that can coexist with what it replaces. A tool that cannot be half-adopted forces a decision; one that can produces a codebase in two styles indefinitely.

What it is

// PHPUnit
final class OrderLineTest extends TestCase
{
    /** @dataProvider invalidQuantities */
    public function testItRejectsANonPositiveQuantity(int $q): void
    {
        $this->expectException(InvalidArgumentException::class);

        new OrderLine(sku: 'ABC-1', quantity: $q);
    }

    public function invalidQuantities(): array
    {
        return [[0], [-1], [PHP_INT_MIN]];
    }
}

// Pest
it('rejects a non-positive quantity', function (int $q) {
    expect(fn () => new OrderLine(sku: 'ABC-1', quantity: $q))
        ->toThrow(InvalidArgumentException::class);
})->with([0, -1, PHP_INT_MIN]);

Fourteen lines becomes four and the second version reads better, which is the entire case for the tool and is not nothing. The runner underneath is PHPUnit, the assertions are PHPUnit’s, and every extension continues to work — so this is a syntax decision rather than a testing-framework migration.

The trial

Question 1: writing

140 tests converted, measured by lines and by time:

  lines        3,204 → 2,102     -34%
  conversion   about 6 hours, mostly mechanical

  where the saving is:
    the class boilerplate            -640 lines
    data providers → ->with()        -410 lines
    setUp() → beforeEach()            -80 lines

  where there is no saving:
    the assertions themselves, which are most of the code
    the fixtures, which are most of the thinking

A third fewer lines is real and it is concentrated in the ceremony rather than in the substance, which is the honest reading — the part of a test that takes time to write is deciding what to assert, and that is unchanged. The ->with() syntax for datasets is the one genuine improvement in writing speed.

Question 2: reading

the argument for:  a test name that is a sentence, and no
                   class wrapper to skip past.

the argument against, which emerged during the trial:

  a Pest test file has no visible structure. 40 tests in
  a file are 40 top-level function calls, and there is no
  navigable outline in an editor.

  higher-order tests — ->group(), ->skip(), ->with()
  chained onto a closure — read well individually and
  produce a file where the control flow is not local.

  a shared beforeEach() in Pest.php applies to a whole
  directory, invisibly, from another file.

verdict: better per test, worse per file.

The global beforeEach in a Pest.php file is the one that produced genuine confusion during the trial: a test failing because of setup defined in a different file with no import and no reference. That is a real readability regression and it is the mechanism the tool encourages.

Question 3: the tooling

  parallel running     works. --parallel, same as before.
  coverage             works. PCOV and Xdebug both.
  the CI reporter      works. JUnit XML is unchanged.
  PHPStan              partial. the closures have no
                       declared $this type, so level 8
                       reports errors inside every test
                       unless the extension is installed —
                       and the extension was at 0.x.
  the IDE              partial. "go to test" and "run this
                       test" work with a plugin; the
                       outline view does not.
  a stack trace        the closure frame is anonymous, so
                       a failure inside a helper is harder
                       to locate.

The static analysis gap is the one that decided it. The suite is analysed at level 8 alongside the application and a tool that requires a 0.x extension to keep that working is a dependency on something that can break — which is exactly the position the team had spent six months getting out of with the PHPStan baseline work.

Question 4: the half-finished migration

both styles run in one suite, which is a feature and is
the risk:

  1,412 tests, of which 140 converted.
  the remaining 1,272 would be converted "when touched",
  which is the same rule that worked for the Vue
  Composition API migration.

  the difference: a component is converted for a reason
  (it is being changed anyway). a test is converted for
  no reason, because the test was already correct.

  so the realistic end state is 300 Pest tests and 1,100
  PHPUnit tests, indefinitely, and two idioms that a new
  person has to learn.

The comparison with the Vue migration is what made this concrete: that one worked because touching a component gives a reason to convert it, and a passing test gives no reason at all. A migration with no forcing function stalls at whatever fraction was done deliberately.

The decision

not adopted, for three reasons in order of weight:

  1  the static analysis gap, which undoes six months of
     work for a syntax preference
  2  the half-migration end state, with no forcing
     function to complete it
  3  the file-level readability regression

and two things taken from the trial anyway:

  the ->with() dataset ergonomics were adopted as a
  convention: a private static method returning an array,
  with STRING KEYS, so a failure names the case

  the 140 converted tests were reverted, and the six hours
  were recorded as the cost of the decision

Recording the six hours as the cost of deciding is the part worth doing, because the alternative is the suggestion returning in six months and the same six hours being spent again. The written outcome — the rubric, the measurements, the verdict — is what makes it a decision that stays decided.

Reverting the converted tests rather than leaving them was deliberate: a hundred and forty tests in the other style is exactly the half-migration state the decision was avoiding. That felt wasteful and was the consistent thing to do.

What this would have cost

Another layer between the tests and the runner, and tooling that half-understands it — which narrows every year and was not narrow enough in September 2022. That is a timing judgement rather than a verdict on the tool, and the written rubric means the next evaluation starts from the three specific gaps rather than from scratch.

The honest caveat is that a team starting a new project would reach a different answer, because there is no existing suite, no half-migration and no six months of static analysis work to protect. Most of the case against this tool is a case against migrating to it, which is a different question from whether it is good.