Sampling a generated diff, and what sample size means

An automated refactor across four hundred files, and a review that cannot read eight thousand lines.

the arithmetic, stated honestly:

  412 files changed by one rule.
  reviewing 40 at random and finding zero problems
  gives roughly 93% confidence that fewer than 5% are
  wrong. it does not give certainty about any file.

  which is fine for a rule whose failures are visible
  (a syntax change), and not fine for one whose
  failures are silent (a semantic change).

so the sample size is not the decision. the rule's
failure mode is.

Splitting the run by rule and reviewing each separately is what makes this tractable, because the confidence needed varies enormously between rules. The rule that rewrote null coalescing had a silent failure mode and was reviewed in full; the one that reformatted docblocks was accepted after a glance.