A suite that runs in four minutes because it stopped touching the database

Fourteen minutes for fourteen hundred tests, on a suite where the great majority assert on pure calculation. The database was involved in eleven hundred of them because the base test case set one up, and the base test case was inherited by everything.

The symptom

final class VatCalculatorTest extends TestCase   // ← AppTestsTestCase
{
    public function testAppliesStandardRate(): void
    {
        $vat = (new VatCalculator())->on(Money::pence(1000), Rate::standard());

        self::assertEquals(Money::pence(200), $vat);
    }
}

// setUp: boots the framework, migrates a database,
// opens a transaction. tearDown: rolls it back.
// the test itself touches nothing.

$ vendor/bin/phpunit --log-junit junit.xml && 
  ./bin/slowest junit.xml | head -5
  0.612s  OrderFlowTest::testFullCheckout
  0.588s  ReportTest::testMonthlySummary
  0.031s  VatCalculatorTest::testAppliesStandardRate
  0.029s  VatCalculatorTest::testAppliesReducedRate
  0.029s  MoneyTest::testAdds

# 31ms for a multiplication. × 1,100 = 34 seconds of
# setup, plus the framework boot that dominates it.

Thirty-one milliseconds is nothing until it is multiplied by eleven hundred, and the framework boot is the larger half of it. The tests were not slow; the scaffolding around each one was.

Why it happens

One base test case is the path of least resistance, and it has to serve the test with the heaviest requirements. Everything inherits the maximum, and nothing in the tooling makes that visible until somebody times it.

The fix

Two suites, and a base class that does nothing

<testsuites>
  <testsuite name="unit">
    <directory>tests/Unit</directory>
  </testsuite>
  <testsuite name="integration">
    <directory>tests/Integration</directory>
    <directory>tests/Feature</directory>
  </testsuite>
</testsuites>

// tests/Unit — extends PHPUnit's TestCase directly
namespace AppTestsUnit;

use PHPUnitFrameworkTestCase;

// tests/Integration — the framework one
namespace AppTestsIntegration;

use AppTestsIntegrationTestCase;   // boots, migrates, wraps

Extending PHPUnit’s own TestCase is the whole mechanism, and it means a unit test cannot accidentally reach a container — the helpers are not there. That constraint is the point rather than a side effect.

The domain layer that never needed persistence

// before: a factory that wrote a row
$order = Order::factory()->create(['total_cents' => 4900]);

// after: a constructor
$order = new Order(
    id: new OrderId('01H8...'),
    lines: [new OrderLine(new Sku('ABC'), 1, Money::pence(4900))],
    placedAt: new DateTimeImmutable('2023-05-18'),
);

The factory existed because the entity could not be constructed without a database — it was an ORM model with a hydration path and no constructor. Giving the domain object a real constructor was the actual change, and the test speed was a consequence rather than the goal.

In-memory repositories, and the contract test

abstract class OrderRepositoryContract extends TestCase
{
    abstract protected function repository(): OrderRepository;

    public function testFindsWhatItSaved(): void
    {
        $repo = $this->repository();
        $repo->save($order = $this->anOrder());

        self::assertEquals($order, $repo->find($order->id));
    }

    public function testReturnsNullForUnknownId(): void { /* ... */ }
    public function testOverwritesOnSecondSave(): void { /* ... */ }
}

// two subclasses: InMemoryOrderRepositoryTest (unit)
// and MysqlOrderRepositoryTest (integration)

The contract test is what makes the fake trustworthy, and without it this refactor is a way of testing against behaviour the database does not have. Eleven assertions, run twice, and the first run of the MySQL subclass found two places where the in-memory version was more forgiving than the real one.

What must stay integrated

stays in the integration suite, deliberately:

  anything asserting a query plan or an index
  anything asserting a database constraint fires
  every migration, run forwards on a clean schema
  the repository implementations themselves
  HTTP tests that go through the router
  anything touching the queue, cache or filesystem

302 tests. 4 minutes 40 seconds. unchanged.

Running the fast one on save

jobs:
  unit:
    steps:
      - run: vendor/bin/phpunit --testsuite=unit
      # no services, no database, 22 seconds

  integration:
    services:
      mysql: { image: mariadb:10.11 }
    steps:
      - run: vendor/bin/phpunit --testsuite=integration

The unit job needing no services is what makes it fast in CI as well as locally — no container to start, no migration to run. Running it on file save locally changed how the domain code gets written more than the CI time did.

Verifying it worked

$ time vendor/bin/phpunit --testsuite=unit
  Tests: 1,112
real    0m22.1s

$ time vendor/bin/phpunit --testsuite=integration
  Tests: 302
real    4m18.4s

$ time vendor/bin/phpunit
real    4m40.9s        # was 14m12s

$ vendor/bin/phpunit --coverage-text | tail -2
  Lines: 74.08%        # was 74.12%

$ vendor/bin/phpunit --testsuite=unit --filter=Repository
  MysqlOrderRepositoryTest: 0 tests   # correctly excluded

Coverage moving by four hundredths of a per cent is the assertion that nothing stopped being tested. The two suites together are ten seconds slower than the integration suite alone, which is the honest accounting — the saving is entirely from not doing the setup eleven hundred times.

What this costs

Two suites, and a rule about which a new test joins. The rule is simple — if it needs a database, it is integration — and it will be got wrong, usually by somebody who adds a repository call to a unit test and moves the file rather than questioning the design.

The in-memory repositories are also code with no production use, kept honest by a contract test that somebody has to remember to extend when the interface grows. A method added to the interface and implemented only in MySQL will pass every unit test and fail in production, which is the failure mode this arrangement introduces.