A flaky test quarantine, and the ticket that goes with it

A test that fails one run in twenty trains the team to re-run the pipeline, and once that habit exists a real failure is re-run too.

/**
 * @group flaky
 * @see https://tracker.internal/ENG-4102 — quarantined 2020-06-14
 */
public function testTheWebhookArrivesInTime(): void { }

// and the pipeline
// vendor/bin/phpunit --exclude-group flaky
// vendor/bin/phpunit --group flaky   ← reported, does not fail the build

Quarantining without a ticket and a date is deletion with extra steps, and the group fills up until nobody remembers what is in it. Running the quarantined tests separately and reporting the result is what keeps them visible — a test that has passed a hundred consecutive times can come back. The habit this protects against is the expensive one: a team that re-runs failures does not have a test suite, it has a slow random number generator.