The test suite that took twelve minutes and now takes ninety seconds

The suite took twelve minutes and nobody ran it. Changes were pushed and CI reported the failure eight minutes later, by which point the author had moved on to something else — which meant the feedback loop was twenty minutes and a context switch rather than twelve minutes.

The symptom

$ time vendor/bin/phpunit
Tests: 1,412, Assertions: 4,102
real	12m04.882s

# and the actual cost, per developer per day:
#   6 pushes × 8 minutes of CI wait
#   plus the context switch on every failure
#
# the local run had been abandoned some time in 2021 and
# nobody could say when.

A suite nobody runs locally is a suite that only catches things after a push, which changes what it is for — it stops being a development tool and becomes a gate. That is a much less valuable thing to have spent four years building.

Why it happens

A suite gets slower one test at a time and nobody notices the increment. There is also no natural moment to look, because the number is only annoying in aggregate and every individual test is defensible.

The fix

Measuring first, which nobody does

$ vendor/bin/phpunit --log-junit junit.xml
$ ./bin/slowest-tests junit.xml | head -4
 41.2s  MigrationSmokeTest::test_all_migrations_run
 28.4s  ImportTest::test_full_catalogue_import
 22.1s  SearchIndexTest::test_reindex_everything
 18.8s  ReportTest::test_annual_summary

$ ./bin/test-time-by-suite junit.xml
  Unit 412 tests 18s | Feature 802 tests 402s | Integration 198 tests 304s

# 88% of the time is in 30% of the tests.

Ranking by duration is thirty seconds of work and it is the step that gets skipped in favour of parallelising everything. Four tests accounted for a hundred and ten seconds between them, and two of those were tests of the test infrastructure rather than of the application.

The migration run, which was two minutes of every run

$ php artisan schema:dump --prune
Database schema dumped successfully.

$ ls database/schema/
mysql-schema.dump

$ time php artisan migrate:fresh --env=testing
Loading stored database schemas: mysql-schema.dump
real	0m4.102s        # was 2m11s

# and the constraint: the dump is loaded only when the
# migrations table is empty, so production is unaffected.

Two hundred and forty migrations replayed by PHP on every fresh database is the single largest fixed cost in most suites, and the dump replaces it with a SQL file the database loads. The trap is the six migrations that inserted data — they are structurally invisible to mysqldump --no-data and their absence produces a schema with no default roles.

$ grep -rln 'DB::table|->insert(' database/migrations/
database/migrations/2015_03_11_seed_roles.php
database/migrations/2016_08_02_backfill_slugs.php
... 4 more

# all six moved to seeders before the prune. finding them
# afterwards means reading them out of git history.

Tests that were integration tests by accident

// 0.4s each, 180 of them: a database round trip to test
// a pure calculation
use RefreshDatabase;

public function testVat(): void
{
    $order = Order::factory()->create(['net_cents' => 10_000]);

    $this->assertSame(2_000, $order->vatCents());
}

// 0.001s: the same assertion, no database
$this->assertSame(2_000, (new VatCalculator())->forNet(10_000));

The second version required extracting the calculation from the model, which is a refactor the test was previously hiding — a method reachable only through a persisted model is a method with a database dependency it does not need. Eighty tests moved this way took seventy seconds off the suite and produced a class with a name.

The rule that identified them: a test using RefreshDatabase and asserting on a value rather than on persistence is testing a calculation. Grepping for that shape found a hundred and forty candidates, of which eighty were worth moving.

Parallel execution, and what it exposed

$ php artisan test --parallel --processes=8

  Tests: 1,412 passed
  Duration: 88.40s

# the first run: 41 failures, all of them shared state.
#   a fixed Redis database        → 8 workers, one cache
#   storage/app/testing/          → one directory
#   a hard-coded queue name       → one queue
#   a fixed port in a stub server → seven collisions

$token = ParallelTesting::token();   // in TestCase::setUp

config([
    'cache.prefix' => "test_{$token}_",
    'database.redis.default.database' => (int) $token,
    'filesystems.disks.local.root' => storage_path("testing/{$token}"),
    'queue.connections.redis.queue' => "test_{$token}",
    'services.stub.port' => 9000 + (int) $token,
]);

Finding every shared resource takes a couple of runs and the failures look exactly like the ordering bugs that randomisation finds, which is confusing when both are being introduced at once. Doing the ordering work first and the parallelism second is the sequence that keeps the failures attributable.

The four slowest tests, individually

41.2s  test_all_migrations_run
       → deleted. schema:dump covers it, and a migration
         that does not run fails every other test.

28.4s  test_full_catalogue_import
       → 40,000 rows reduced to 200. the assertion was
         about correctness, not volume. a separate
         nightly job covers the volume case.

22.1s  test_reindex_everything
       → moved to the nightly suite. it tests a command
         that runs weekly.

18.8s  test_annual_summary
       → the fixture generated 12 months of orders in a
         loop. replaced with a seeded SQL file: 0.9s.

Three of the four were tests whose value did not depend on running on every push, and moving them to a nightly suite is not a loss — it is putting them where their cost is affordable. Deleting the migration test outright was the right call and needed saying out loud, because deleting a test always feels like a regression.

Verifying it worked

$ time php artisan test --parallel
Tests: 1,404 passed
real	1m28.402s        # was 12m04

$ ./bin/test-time-by-suite junit.xml
  Unit          492 tests     6s
  Feature       742 tests    58s
  Integration   170 tests    24s

# five consecutive randomised runs, all green
$ for i in 1 2 3 4 5; do php artisan test --parallel | tail -1; done

# and the number that actually mattered:
#   local runs per day, measured from a git hook: 0 → 31

The local run count is the outcome and it is the only one that changes behaviour — a suite people run before pushing catches the failure in the editor rather than eight minutes later in a browser tab. Instrumenting it with a git hook was two lines and made the case for the work in a way the wall-clock number did not.

What this costs

Eight databases, eight cache prefixes and eight temporary directories, plus a per-worker configuration block that a ninth shared resource will silently break. The failure mode when it does is a flaky test that looks exactly like the ordering problem this work was meant to remove, which is why the configuration lives in one place with a comment explaining what belongs there.

The nightly suite is the other cost and it is the one that will rot. Three tests moved out of the main run are three tests that fail on a Tuesday morning against a change from the previous Thursday, and the connection is not obvious. Failing the nightly loudly into a channel somebody reads is the mitigation, and it is weaker than a blocked pull request.