Mutation testing, and the tests that assert nothing

The pricing bug was in a method with 100% line coverage. Two tests called it, both asserted that it returned a number, and neither checked which number — so inverting the discount comparison broke nothing anybody could see. The coverage report had been green for two years and had never been measuring what everybody thought.

The symptom

public function discountFor(Customer $c, Money $total): Money
{
    if ($total->cents() >= 5000 && $c->isPremium()) {
        return $total->percent(10);
    }

    return Money::zero($total->currency());
}

// the tests, both of which pass with >= changed to >
public function testItReturnsMoney(): void
{
    $this->assertInstanceOf(Money::class,
        $this->calc->discountFor($this->premium(), Money::gbp(5000)));
}

public function testItDoesNotThrow(): void
{
    $this->calc->discountFor($this->standard(), Money::gbp(100));
    $this->addToAssertionCount(1);
}

Both lines are covered, both tests pass, and the boundary — which is where the entire business rule lives — is untested. The second test asserts nothing at all and incremented the assertion counter to avoid a risky-test warning, which is a thing somebody did on purpose.

Why it happens

Coverage instruments which lines executed. It has no opinion about whether anything was checked, so a test that calls a method and asserts its return type covers every line in it — and the number goes up, which is what the number is for.

The incentive follows: a team measured on coverage writes tests that execute code, and a team measured on nothing writes fewer tests. Neither produces a suite that catches a changed comparison operator.

The fix

Change the code deliberately and see whether anything notices

$ vendor/bin/infection --threads=4 --min-msi=50

412 mutations generated:
     251 killed
     118 not covered by tests
      43 covered but NOT detected     ← the finding

  MSI: 60%   Mutation Code Coverage: 71%   Covered Code MSI: 85%

The tool changes the source one mutation at a time, runs the tests, and reverts. A mutant that is killed means some test failed, which means something was actually being checked. A mutant that escapes means the change made no test fail — which is the finding.

MSI                     over ALL mutants
Mutation Code Coverage  over the ones tests reach at all
Covered Code MSI        over those — and it is the useful one

of the code the tests execute, how much do they verify?
85%, against a line coverage of 94%. that gap is the finding.

Reading the escaped mutants, which is where the finding is

--- Original
+++ New
@@ @@
     public function discountFor(Customer $c, Money $total): Money
     {
-        if ($total->cents() >= 5000 && $c->isPremium()) {
+        if ($total->cents() > 5000 && $c->isPremium()) {

--- Original
+++ New
@@ @@
-        return $total->percent(10);
+        return $total->percent(11);

Boundary mutants are the most valuable finding by a wide margin. >= becoming > escaping means nothing tests the exact threshold, which is where every off-by-one in a pricing rule comes from — and the second mutant shows that nothing checks the discount amount either.

The score is a summary and the list is the product. Reading the escaped mutants for one class teaches more about the suite than the percentage ever will, and chasing the number instead produces tests written to kill mutants — which is a worse suite that scores better.

// what the findings asked for
$this->assertEquals(Money::gbp(500),
    $this->calc->discountFor($this->premium(), Money::gbp(5000)));

$this->assertTrue(
    $this->calc->discountFor($this->premium(), Money::gbp(4999))->isZero());

The ones that are equivalent, and why the score is never 100

// a mutant nothing can kill: the two are identical
for ($i = 0; $i < $count; $i++)        // original
for ($i = 0; $i <= $count - 1; $i++)   // mutant

// and the suppression, which is a claim worth reviewing
/**
 * @infection-ignore-all
 */

// or, narrowly, per mutator
// infection.json: "mutators": { "LessThanOrEqualTo": { "ignore": [...] } }

Some mutations produce code that behaves identically, so no test can kill them — and they are indistinguishable from real findings without reading each one. A target of 100% is therefore wrong and chasing it produces suppressions rather than tests. Somewhere between 60 and 80 is defensible for a mature codebase, and the direction matters more than the number.

Each suppression is a claim that a mutant is equivalent, and some of those claims are wrong. Reviewing them like any other assertion is the discipline that keeps the tool honest, and a suppression on a whole class is almost always somebody giving up rather than a genuine equivalence.

Making it affordable

# the full run, on this codebase
$ time vendor/bin/infection --threads=8
real    2h 41m

# the diff-only run, which is what goes in CI
$ vendor/bin/infection 
    --git-diff-filter=AM 
    --git-diff-base=origin/main 
    --min-msi=70 
    --logger-github
real    2m 12s

# 41 mutants instead of 4,120.

A three-hour run happens nightly and nobody reads it. A two-minute run happens on every pull request and lands as an annotation on the changed lines, which puts the finding where the author is already looking.

Applying the threshold only to new code is the same ratchet as a coverage baseline: the existing score is what it is, and anything added has to meet the bar. The nightly full run is still worth having for the trend, and nobody needs to read it every day.

Verifying it worked

# the mutant that started this, after the two new tests
$ vendor/bin/infection --filter=src/Pricing/DiscountCalculator.php

12 mutations were generated:
      12 mutants were killed
       0 covered mutants were not detected

  Covered Code MSI: 100%

# and the suite-wide picture, six weeks later
#   line coverage:      94%  →  93%   (barely moved)
#   covered code MSI:   85%  →  91%
#   assertions:      3,891  →  4,402

Line coverage going down slightly while the mutation score went up is the outcome that says this worked: the new tests assert rather than execute, and two of the old ones that executed without asserting were deleted. A metric that improves while coverage falls is exactly the signal coverage cannot give.

Five hundred additional assertions across roughly the same number of tests is the other number, and it is the one to quote — the suite did not grow, it started checking things.

What this costs

A very slow tool and a metric that is easy to game. A full run is hours, which means the feedback loop is a day unless it is restricted to the diff — and a team that decides to chase the score will write tests that kill mutants without testing behaviour, which is possible and produces a suite that is worse and scores better. The defence is that the escaped mutants are read rather than counted, and that is a review habit rather than a configuration.

The honest limitation is that it says nothing about the tests you have not written at all. A class with no tests produces mutants that are “not covered” rather than “escaped”, and the score for covered code can be excellent while most of the application is untested. Reading the three numbers together rather than the headline one is what keeps that visible, and the headline number is the one that ends up on a dashboard.