The flaky test quarantine was introduced in 2023 with a deadline attached to every entry, which turned a list of eleven into a list of two within a month. One of the two survived three deadline extensions and was still skipped in 2026, and it covered payment capture.
The symptom
#[Group('flaky')]
#[Ticket('ENG-412', until: '2023-11-01')]
#[Ticket('ENG-412', until: '2024-06-01')]
#[Ticket('ENG-412', until: '2025-03-01')]
#[Ticket('ENG-412', until: '2026-01-01')]
public function testAPaymentIsCapturedExactlyOnce(): void
{
$this->service->capture($this->order());
$this->service->capture($this->order()); // a redelivery
self::assertCount(1, Capture::all());
self::assertDatabaseHas('payments', ['status' => 'captured']);
Queue::assertPushed(SendReceipt::class, 1);
}Four annotations is four decisions to defer, each of them made by somebody looking at a failing pipeline and choosing the option that unblocked it. A quarantine deadline that can be extended is a policy with an escape hatch, and the escape hatch is used at exactly the moment nobody has time to do otherwise.
Why it happens
A flaky test is a cost paid by whoever is unlucky and a fix is a cost paid by whoever volunteers. Extending the deadline moves the cost to a future person, which is the rational individual choice every time.
The fix
Splitting one test into three
// three assertions across two systems, in one test.
// splitting is diagnosis rather than a fix.
public function testASecondCaptureDoesNotCreateASecondRecord(): void
public function testTheOrderIsMarkedPaid(): void
public function testAReceiptIsQueuedExactlyOnce(): void
// 1,000 runs of each:
// the first two: 0 failures
// the third: 84 failures (8.4%)The combined test failed eight per cent of the time and the failure could have been any of three things, which is why three years of looking at it produced nothing. Splitting identifies which assertion is unreliable and takes twenty minutes.
Three causes, discovered in order
1 the clock. the test travelled forward 30 seconds
and asserted on a window that is 30 seconds,
inclusive at one end. under parallel load the two
now() calls could straddle it.
→ an injected clock. failure rate 8.4% → 3.1%.
2 the queue fake. Queue::assertPushed counts pushes
in the current process, and the capture dispatches
from inside a transaction with an after-commit
callback. under some orderings the assertion ran
before the commit.
→ assert after an explicit commit. 3.1% → 0.9%.
3 and the ninth of a per cent.Two fixes and the rate does not reach zero, which is the point at which most investigations stop — a test failing one run in a hundred is annoying rather than blocking. The remaining nine tenths of a per cent is where the actual bug was.
The third cause, which was in the code
// the claim, and the dispatch
DB::transaction(function () use ($message) {
$claimed = DB::table('processed_messages')->insertOrIgnore([
'handler' => self::class,
'message_id' => $message->id,
]);
if ($claimed === 0) { return; }
$this->capture($message);
SendReceipt::dispatch($message->orderId); // ← inside
});
// dispatch() inside a transaction, without
// afterCommit, means the job can be picked up by a
// worker BEFORE the transaction commits — and the
// worker reads a payment row that does not exist yet.which produces, in production:
a receipt job that finds no payment, fails, retries
three times over ten minutes, and succeeds on the
second or third attempt once the transaction has
committed.
so the receipt is sent, late, and the failure is a
transient in a log nobody reads.
frequency, from three years of the failed-jobs
table: 41 occurrences. one of which exhausted its
retries in 2024 and was recorded as a support
anomaly.A race between a transaction and a queue worker is the classic version of this and it had been in production since 2023, producing a late receipt forty-one times and one that never arrived. The test had been failing for the right reason for three years.
The fix, which is one argument
SendReceipt::dispatch($message->orderId)->afterCommit();
// or, at the connection level, so it is the default
// rather than a thing each dispatch remembers:
'redis' => [
'driver' => 'redis',
'after_commit' => true,
],Setting it at the connection level is the version that survives somebody forgetting, and it changes behaviour for every dispatch in the application — which needed checking. Two dispatches genuinely wanted the old behaviour and are now explicit about it.
The quarantine policy, revised
before: a deadline, extendable by anybody, silently.
after:
an entry may be extended once, by anybody
a second extension requires the reason to be written
in the ticket
a third is not possible — the test is either fixed or
deleted, and deleting it requires deleting or
justifying the code it covers
and the weekly job now reports the AGE of each entry
rather than whether it is past its deadline, because
the age is the number that was never visible.The third extension being impossible is the only part with teeth, and it forces the question that three years of extensions avoided — is this test worth having. For a test covering payment capture the answer was obviously yes, and nobody had been made to answer it.
Verifying it worked
$ for i in $(seq 1 1000); do
vendor/bin/phpunit --filter testAReceiptIsQueued
>/dev/null 2>&1 || echo "failed on $i"
done
# 41 minutes, no failures
# which bounds the rate under about 0.3% at 95%
# confidence. the original was 8.4%.
$ mysql -e "SELECT COUNT(*) FROM failed_jobs
WHERE payload LIKE '%SendReceipt%'
AND failed_at > '2026-07-01'"
0 # was ~14 a year
$ ./bin/quarantine-report
entries: 0A thousand clean runs bounds the failure rate rather than eliminating it, and stating the bound is the difference between evidence and a feeling. It matters here because the fix was a genuine race — had it been a longer timeout, a thousand runs would have proved considerably less.
What this costs
A quarantine policy that now has to be enforced, by a weekly report that somebody reads. The three-extension limit is a rule and the enforcement is a person noticing, which is the same mechanism that failed for three years — the difference is that the age is now visible rather than the deadline.
The connection-level after_commit is also a behaviour change to every dispatch in the application, made to fix one. Two places wanted the old behaviour and are now explicit; a third that nobody found would be a job that no longer runs when its transaction rolls back, which is almost certainly correct and is a change nobody asked for.