Mutation testing for one directory, and what it found

The pricing code had ninety-four per cent line coverage and shipped a bug that charged a discount twice at one boundary condition. Coverage measures which lines executed, and the test that executed those lines asserted only that nothing threw.

The symptom

public function testCalculatesTieredDiscount(): void
{
    $result = $this->calculator->apply(
        Money::pence(10000),
        Tier::from('gold'),
    );

    self::assertInstanceOf(Money::class, $result);   // ← this
}

// 100% of the method's lines executed. one assertion,
// about the return type.

the shipped bug:

  if ($total->cents > self::TIER_THRESHOLD) { ... }

  should have been >=. an order at exactly £100.00 fell
  into the wrong branch and received both the tier
  discount and the promotional one.

  affected orders before it was found: 412
  value: £1,880

Why it happens

Coverage answers “did this line run”, which is a necessary condition for a test to catch a bug in that line and nowhere near sufficient. A test with no meaningful assertion produces the same coverage as a thorough one.

The fix

What a mutant is

the tool changes the code, then runs the tests.

  original    if ($total->cents > self::THRESHOLD)
  mutant      if ($total->cents >= self::THRESHOLD)

tests pass → the mutant SURVIVED: no test distinguishes
the two behaviours. tests fail → it was KILLED.

the three operators that find the most:
  comparison   > ↔ >=, < ↔ <=, == ↔ !=
  arithmetic   + ↔ -, * ↔ /
  return       return $x → return null

One directory, because the whole thing takes nine hours

{
  "source": { "directories": ["src/Domain/Pricing"] },
  "mutators": { "@default": true },
  "minMsi": 85,
  "minCoveredMsi": 90,
  "timeout": 10
}

$ vendor/bin/infection --only-covered --threads=4
  Mutation Score Indicator (MSI):    71%
  Mutation Code Coverage:            94%
  Covered Code MSI:                  76%
  Mutations: 412  Killed: 293  Escaped: 94  Errors: 25

Time: 11m 20s      # the whole codebase: 9h 10m

Eleven minutes for one directory against nine hours for everything is the difference between a tool that gets used and one that gets configured and forgotten. Choosing the directory where a bug is expensive rather than the one where coverage is lowest is the decision that makes this worthwhile.

The ninety-four survivors, triaged

  41  equivalent mutants — the change does not alter
      behaviour → ignored, some with a config entry
  38  mutations of things that do not matter, such as a
      log message's severity → excluded by config
  11  untested branches in error handling → 4 tests
      added; 7 judged not worth it, and recorded
   4  REAL GAPS in the calculation itself → one of them
      was the shipped bug

// the test that kills it
#[DataProvider('tierBoundaries')]
public function testTierBoundary(int $pence, string $expected): void
{
    $result = $this->calculator->apply(Money::pence($pence), Tier::gold());

    self::assertSame($expected, $result->describe());
}

public static function tierBoundaries(): array
{
    return [
        'below'  => [ 9999, 'promotional only'],
        'exact'  => [10000, 'tier only'],      // ← the bug
        'above'  => [10001, 'tier only'],
    ];
}

Three cases around one boundary, which is the shape of test that kills a comparison mutant and the shape nobody writes without being prompted. The exact-value case is the one that was missing and it is always the one that is missing.

In CI, on a schedule

on:
  schedule:
    - cron: '0 2 * * 2'          # Tuesday, overnight
  pull_request:
    paths: ['src/Domain/Pricing/**']

jobs:
  mutation:
    steps:
      - run: |
          vendor/bin/infection --only-covered 
            --min-msi=85 --min-covered-msi=90 
            --logger-github

Running it on pull requests that touch the directory and weekly otherwise is the compromise that keeps it useful without adding eleven minutes to every build. The threshold is a ratchet: it started at 71 and was raised as the score improved, which is the only way a quality gate gets adopted.

The metric that is easy to game

ways to raise the score without improving anything:

  narrow the source directories until only
    well-tested code is measured
  add mutators to the ignore list
  assert on internals so that any change fails

the last one is the dangerous one: a test asserting on
every intermediate value kills every mutant and makes
the code impossible to refactor.

the defence is that the score is not a target for
anybody. it is a report that produced four tests.

Verifying it worked

$ vendor/bin/infection --only-covered
  MSI:                88%       # was 71%
  Covered Code MSI:   94%       # was 76%
  Escaped:            41        # was 94, all triaged

$ vendor/bin/phpunit --coverage-text --filter Pricing
  Lines: 94.2%                  # unchanged

# the deliberate check: reintroduce the shipped bug
$ sed -i 's/>= self::TIER/> self::TIER/' src/Domain/Pricing/Calculator.php
$ vendor/bin/phpunit --filter TierBoundary
  FAILED: 1 of 3 — 'exact' expected 'tier only'
# which is what a test suite should have done in June

Reintroducing the original bug and confirming the suite now catches it is the assertion worth making, and it takes thirty seconds. Coverage being unchanged at ninety-four per cent while the mutation score moved by seventeen points is the whole argument for the exercise.

What this costs

A slow job and a metric that is easy to game. Eleven minutes on one directory is affordable and the score is a number somebody will eventually be asked to improve, at which point the cheapest route is narrowing the scope rather than writing tests.

The triage is also the expensive part and it does not go away. Forty-one surviving mutants remain, most of them equivalent, and each new run requires somebody to check whether a new survivor is one of those or something real — which is judgement, not automation.