A year of an assistant in the editor, measured

A year of a model-backed assistant in the editor, across three people and one codebase. Everything written about this is either a demonstration of it working or a complaint about it failing, and both are selected examples — this is an attempt to say something specific, with the measurement problems stated rather than hidden.

The symptom

the question that started this: is it helping?

what was available to answer it:

  acceptance rate      41% of suggestions accepted
  edit distance after
    acceptance         median 18 characters changed
  lines suggested      ~4,100 a week

and what none of those tell you:

  whether the accepted code is correct
  whether it would have been written better without
    the suggestion
  whether time saved writing was spent reviewing
  whether reading a rejected suggestion cost time

A forty-one per cent acceptance rate is compatible with a tool producing correct code and with one producing plausible code, and nothing in the telemetry distinguishes them. The counterfactual — what would have been written otherwise — is not available at any price, which is the whole measurement problem.

Why measuring is hard

The unit of work is not comparable between tasks, the same task is never performed twice, and the person performing it changes as they use the tool. Any before-and-after comparison is confounded by the codebase changing, the team changing and the models changing, all within the measurement window.

The fix

What can honestly be counted

three things, all of them proxies:

  1  review findings on generated code, categorised
     — requires knowing which code was generated,
       which requires people to mark it

  2  defect rate by origin, over time
     — a lagging indicator, confounded by everything

  3  a controlled comparison on one task type
     — writing tests for existing code, where the
       task IS comparable

we did all three. only the third produced a number
anybody should believe.

The controlled comparison

task: write a test suite for an existing class with no
tests. 22 classes, selected by coverage, assigned
alternately.

  with assistance     11 classes, median 18 minutes
  without             11 classes, median 31 minutes

  tests produced      with: median 9    without: 6
  mutation score of
    the resulting
    tests             with: 61%   without: 74%

faster, more tests, and the tests are worse.

the extra tests are the ones asserting that a
constructor assigns its arguments.

More tests and a lower mutation score is the finding, and it is the most useful number in the whole exercise. Generated suites are padded with assertions that execute code without checking anything, which raises coverage and lowers the actual verification — and a coverage-based quality gate rewards exactly this.

Where it is straightforwardly good

  tests for existing code       with the caveat above
  the shape of a migration      expand-and-contract
                                scaffolding, which is
                                mechanical and
                                error-prone by hand
  a first draft of a data
    transformation              CSV column mapping,
                                and it is verifiable
                                immediately
  boilerplate with a clear
    specification               a PSR-18 decorator,
                                a value object with
                                six properties
  explaining unfamiliar code    the language does not
                                matter and the answer
                                is checkable

Where it is confidently wrong

// suggested, and idiomatic PHP
public function getTotal(): float
{
    return $this->totalMinorUnits / 100;
}

// this codebase, since 2019: money is minor units in
// an integer. a float total is the bug we spent a
// quarter removing in 2021.

// suggested, and idiomatic for the framework
$order = Order::find($id);

// this codebase: repositories, and Order is a domain
// object rather than a model.

Both suggestions are what most code in this language looks like, which is why they are dangerous — they read as correct to a reviewer who is skimming, because they are correct almost everywhere else. A conventions file the tool reads improved the second case substantially and the first case not at all.

The security review

// 1. a query built by concatenation, in a codebase
//    where every other query is bound
$sql = "SELECT * FROM orders WHERE reference = '$ref'";

// 2. a signature compared with ===
if ($expected === $provided) { /* timing-sensitive */ }

// 3. an error response echoing the exception message
return response()->json(['error' => $e->getMessage()], 500);

three findings in six months, all of them in code
that was accepted and caught in review.

what they have in common: all three are patterns that
appear constantly in public code, and all three would
pass a distracted review because they look ordinary.

the second is the one that worries me most.
hash_equals is not visually distinctive and its
ABSENCE is not something a reviewer scans for.

None of these reached production and all three were accepted at the point of suggestion, which is the number that matters — the review caught them and the review is a person who was paying attention that day. That is the same defence as before the tool existed and it is now defending against a higher volume.

The habit it changes

before: you write code, then somebody reviews it.
after: you review code, then somebody reviews it.

which sounds like more review and is not, because the
first pass is performed by somebody who did not think
through the problem — they are checking whether the
answer looks right rather than whether it is right.

the specific failure: accepting a suggestion that is
plausible, then reasoning forwards from it, and never
considering the approach it foreclosed.

This is the effect I am least able to measure and most concerned about. A suggestion arrives before the problem has been thought through, and evaluating it is a different cognitive act from solving the problem — the suggestion becomes the anchor, and the design that was not suggested is never considered.

What we do not let it near

## Where model assistance is not used

- authentication, authorisation, session handling
- anything that constructs a query
- cryptographic operations, including signature
  comparison
- database migrations
- the deploy pipeline

## Why
The first three because a plausible-looking mistake is
a vulnerability. The last two because a mistake is not
reversible by a redeploy.

## Review: 2026-03

Two categories with a stated reason each, rather than a general caution nobody can apply. It is unenforceable — there is no mechanism, only a norm checked at review — and the review date is what stops it becoming folklore that outlives its reasoning.

The disagreement

the agreed facts, after a year:

  faster at scaffolding and at tests for existing code
  confidently wrong about this codebase's conventions
  review takes longer per line, and there are more
    lines
  net output: unmeasured, and probably unmeasurable

the disagreement:

  one position — the review cost is worth the speed,
    and the conventions problem is solvable
  the other — reviewing code nobody wrote erodes the
    understanding that makes review possible, and
    that cost compounds

both follow from the facts. neither is refutable with
more data.

A disagreement that survives agreement on the evidence is about values rather than facts, and treating it as an empirical question wastes everybody’s time. What was settled instead is scope — where it is used and where it is not — which two people with different views can both accept.

Verifying it worked

$ ./bin/review-findings --since=6m --origin=generated
  total findings:              88
  of which security:            3
  of which convention:         41
  of which correctness:        18
  of which style:              26

$ ./bin/review-findings --since=6m --origin=human
  total findings:              62
  of which security:            1
  of which convention:          8

$ ./bin/defect-rate --since=12m --by-quarter
  Q4 2024  0.41   Q1 2025  0.38
  Q2 2025  0.44   Q3 2025  0.39
  # per 1,000 lines changed. no signal.

The convention findings being five times higher on generated code is the clearest signal in the data and it is exactly what the controlled comparison predicted. The defect rate showing no movement across four quarters is the honest headline — a year in, the thing everybody wants to know remains unanswered.

What this costs

Reviewing more code than before, with the same attention available. Volume went up, findings per line went up, and the reviewer is the same person having the same day — which is a load-bearing assumption that nothing in this arrangement protects.

A skill that atrophies, and the open question of which one. The obvious candidate is writing code from scratch, which matters less than it sounds; the one I would actually worry about is the habit of sitting with a problem before producing an answer, because a suggestion arrives before that has happened.

And the measurement problem does not go away. A year of data produced one number worth believing — from a controlled comparison on one task type — and no signal at all on the question anybody is asking. Anybody claiming a percentage improvement in engineering productivity from this is measuring something else and has not said what.