A thousand runs, and what that proves

A test run a thousand times to establish that a flake is fixed, and what that number actually supports.

$ for i in $(seq 1 1000); do
    vendor/bin/phpunit --filter testReceiptIsQueued 
      >/dev/null 2>&1 || echo "failed on run $i"
  done
# 41 minutes, no failures

# what that supports: at 95% confidence, the failure
# rate is under about 0.3%.
#
# the original rate was ~8%. so this is strong evidence
# it is fixed and is not proof, and the difference
# matters if the fix was a timing change rather than a
# cause.

A thousand clean runs bounds the failure rate rather than eliminating it, and stating the bound is the difference between evidence and a feeling. It mattered here because the fix was a genuine race condition — had it been a longer timeout, a thousand runs would have proved considerably less.