The integration suite passed on every machine and failed roughly one CI run in six, always on a different test. Re-running it made it pass, so the team had learned to re-run it, which meant the suite was reporting nothing and costing four minutes.
The symptom
$ vendor/bin/phpunit --testsuite=integration
Tests: 214, Failures: 0
$ vendor/bin/phpunit --testsuite=integration --order-by=random
Tests: 214, Failures: 3
1) OrderExportTest::testIncludesArchivedOrders
Failed asserting that 0 matches expected 4.
2) CustomerTierTest::testGoldCustomersGetFreeShipping
Failed asserting that false is true.
# both pass alone. both depend on rows created elsewhere.A suite that only passes in one order is testing the order. The three failures were all the same shape: a test asserting on data another test happened to leave behind.
Why it happens
A test that needs four archived orders finds four already present, passes, and never gets a fixture written for it. The dependency is created by omission rather than by decision, which is why nobody can point at when it was introduced.
The fix
Randomising the order, permanently
<phpunit executionOrder="random"
resolveDependencies="true"
cacheResult="true"
cacheResultFile=".phpunit.cache">
<!-- the seed is printed on every run:
Randomized with seed 1621899402
and reproduces exactly:
--order-by=random --random-order-seed=1621899402 -->The seed being reproducible is what makes this workable rather than infuriating. A random failure that cannot be reproduced is worse than a hidden dependency, and the seed printed on every run turns it into a normal debugging session.
Locally the useful setting is defects, which runs the previously failing tests first — it makes the fix loop fast and does not exercise the ordering. Both settings in the same file, chosen by a CI environment variable, is the arrangement that lasted.
A transaction per test, and what it does not cover
use IlluminateFoundationTestingRefreshDatabase;
final class OrderExportTest extends TestCase
{
use RefreshDatabase; // begin a transaction, roll back after
public function testIncludesArchivedOrders(): void
{
Order::factory()->count(4)->archived()->create();
$this->assertCount(4, (new OrderExport())->archived());
}
}what the transaction does NOT cover:
a second connection — a browser driver hitting the app
over HTTP sees a different session and none of the
uncommitted fixtures. the page renders empty.
DDL — MySQL commits implicitly on CREATE TABLE, so a
test that runs a migration ends the transaction and
everything after it leaks.
after-commit callbacks — they never fire, so anything
dispatched from one is silently untested.The second-connection case produces the most confusing failure of the three, because the test is asserting on a rendered page that is genuinely empty and every fixture assertion passes. Browser tests need a different strategy — truncation, or a migrated database — at a real speed cost.
The after-commit gap is the subtle one: a test passes, the behaviour it was written for never runs, and the coverage report says the line is covered. Anything dispatched from afterCommit needs a test that commits, which means opting that class out of the transaction wrapper.
Fixtures that state what they need
// implicit: passes only if something else created them
public function testGoldCustomersGetFreeShipping(): void
{
$customer = Customer::where('tier', 'gold')->first();
$this->assertTrue($this->shipping->isFreeFor($customer));
}
// explicit: states its own world
public function testGoldCustomersGetFreeShipping(): void
{
$customer = Customer::factory()->gold()->create();
$this->assertTrue($this->shipping->isFreeFor($customer));
}The rule that removes the whole class is that a test never queries for data it did not create. A ->first() in a test is a dependency on whatever happens to be there, and grepping for the pattern found nineteen of them.
Factory states are what make the explicit version short enough that people write it. Without gold(), the setup is four lines of attribute assignment and the implicit version starts looking reasonable again.
Parallel workers, which need a database each
$ php artisan test --parallel --processes=8
# app_test_1 .. app_test_8, created once and reused
# and the setup cost, which decides whether it is worth it:
# 8 × migrate 6m10s
# 8 × schema dump 24sThe setup cost can exceed the time saved, which is why the schema dump matters more here than anywhere — eight full migration runs against two hundred migration files is six minutes of a four-minute suite. Loading a dump makes the parallelism worth having.
// anything shared outside the database needs the worker
// number in it, and finding them all takes a few runs
$token = ParallelTesting::token(); // 1..8, or null
config([
'cache.prefix' => "test_{$token}_",
'filesystems.disks.local.root' => storage_path("testing/{$token}"),
'database.redis.default.database' => (int) $token,
]);Redis is the one that catches people, because a shared cache across eight workers produces failures that look exactly like the ordering problem the parallelism was introduced alongside. Separating the database index per worker is one line and removes it entirely.
Verifying it worked
$ for i in 1 2 3 4 5; do
> vendor/bin/phpunit --order-by=random | tail -1
> done
OK (214 tests, 611 assertions)
OK (214 tests, 611 assertions)
OK (214 tests, 611 assertions)
OK (214 tests, 611 assertions)
OK (214 tests, 611 assertions)
$ grep -rn '::first()|::latest()->first()' tests/ | wc -l
0
$ php artisan test --parallel
Tests: 214 passed Duration: 48.20s # was 4m12Five consecutive randomised runs is the assertion, and one is not enough for a failure that appears one time in six. Making that loop a CI step on the default branch is what stops the dependency reappearing.
Four minutes to forty-eight seconds is mostly the parallelism rather than the ordering fix, and the two had to happen together — parallel workers against a suite with ordering dependencies fails constantly rather than occasionally.
What this costs
Every test now creates its own fixtures, which is more setup code and a slower suite per test — the implicit version was free because somebody else had paid for it. Factory states absorb most of that and the remainder is a real trade for a suite whose failures mean something.
Eight databases and eight cache prefixes is more moving parts in CI, and the failure mode when one is misconfigured is a flaky test that looks exactly like the problem being fixed. Keeping the per-worker configuration in one place, with a comment saying why, is what makes it debuggable — the alternative is discovering a shared temporary directory eighteen months later.