A test suite that runs on every commit, in ninety seconds

The suite has been at four minutes forty since 2023, which is fast enough that nobody complains and slow enough that people batch their work around it. Getting it to ninety seconds was not about the pipeline — it was about whether running the tests is a decision somebody makes.

The symptom

  unit         1,112 tests    22s
  integration    302 tests   4m 18s
  ─────────────────────────────────
  total        1,414         4m 40s

and the observable behaviour:

  local runs per developer per day    4
  pushes with a failing test          11%
  median time from writing a bug to
    seeing it fail                    ~14 minutes

Four local runs a day means people run the suite before pushing rather than while working, which is what a four-minute wait produces. Eleven per cent of pushes failing is the consequence — the feedback arrives after the work is finished rather than during it.

Why it happens

There is a threshold below which running the suite stops being a decision and becomes a reflex, and it is somewhere under two minutes. Above it people batch; below it they do not, and the difference is not proportional to the time saved.

The fix

Where the four minutes went

  188 migrations, per run          3m 41s
  302 integration tests            37s
  1,112 unit tests                 22s

the migrations are 79% of the suite and are not a test.

A schema dump rather than migrations

# once, and committed
php artisan schema:dump --prune

# per run
mysql -h127.0.0.1 app_test < database/schema.sql

# 3m 41s → 9s

# and the check that stops it going stale
php artisan schema:dump --prune
git diff --exit-code database/schema.sql 
  || { echo 'run php artisan schema:dump --prune'; exit 1; }

The staleness check is what makes this safe — a migration added without regenerating the dump is silently absent from every test run, which is the worst possible failure because the tests still pass. The --prune flag deletes the migration files that are now represented in the dump, which is the part that requires trusting the dump.

A database per process

// paratest gives each process a TEST_TOKEN
'database' => env('DB_DATABASE', 'app_test')
    . (env('TEST_TOKEN') ? '_' . env('TEST_TOKEN') : ''),

// app_test_1 .. app_test_8, created once in CI, each
// loaded from the same dump.

why a database per process rather than a transaction
per test:

  a transaction isolates a test from its OWN writes.
  it does not isolate process 1 from process 2, and
  two processes inserting the same unique email is a
  duplicate key error in a test that is correct.

  8 databases × 9s of schema load = 72s, in parallel,
  which is 9s of wall clock.

The tests that could not be parallelised

#[Group('serial')]
final class ExportFileTest extends TestCase
{
    // writes to storage/app/exports/monthly.csv — a
    // fixed path in the application's configuration,
    // not in the test
}

// and the pipeline runs two commands:
//   vendor/bin/paratest -p8 --exclude-group=serial
//   vendor/bin/phpunit --group=serial

Eleven tests in the serial group take fourteen seconds, which is a rounding error, and making the fixed path configurable would have been a change to the application to suit the tests. Two runs is more honest than pretending the constraint is in the test.

The flake that only appeared under parallelism

// failed roughly one run in twelve, and had passed
// ten thousand times serially
$order = Order::factory()->create(['placed_at' => now()]);

$this->travelTo(now()->addSeconds(30));

self::assertTrue($order->isWithinCancellationWindow());

// the window is 30 seconds, inclusive at one end.
// under load, the two now() calls could straddle it.

Serial execution made the timing deterministic by accident, and parallelism removed the accident rather than introducing the bug. The fix is an injected clock — a test that depends on wall-clock elapsed time has a race in it regardless of how it is run, and the race was always there.

What runs on save, and what runs on push

  on save (a watcher)    the unit suite for the
                         changed directory. 2-4s.
  on commit (a hook)     the full unit suite. 22s.
  on push (CI)           everything, parallel. 90s.
  nightly                everything serially, to catch
                         a test that only passes
                         because of parallel isolation

the nightly serial run is the check on the check, and
it has found one test that depended on another
process's database being absent.

Verifying it worked

$ time ./bin/test
real    1m28s              # was 4m 40s

$ ./bin/flake-report --since=30d
  runs: 412   flaky failures: 2   rate: 0.5%

# and the behaviour change, over a month
  local runs per developer per day   4 → 22
  pushes with a failing test         11% → 3%
  time from writing a bug to seeing
    it fail                          14m → 40s
  commits per day                    +18%

The commit rate going up is the second-order effect and the one that suggests the four-minute suite had been shaping the work rather than merely verifying it. People were batching changes to amortise the wait, which is a design decision the tooling was making on their behalf.

What this costs

Parallel tests fail in ways serial tests do not, and the flake rate went from zero to half a per cent. Two flaky failures a month is a small number and it is not zero, and each one erodes trust in the suite by more than a legitimate failure does.

The schema dump is also a generated artefact under version control that must be regenerated with every migration, enforced by a check that somebody will disable when it fails at an inconvenient moment. A stale dump means tests running against a schema that no longer exists, passing, and telling you nothing.