We had generated the API document from annotations since 2020, which solved the problem of documentation drifting behind the code. It did not solve the problem of the code drifting away from what clients expected, because a document generated from the implementation agrees with the implementation by construction.
The symptom
$ git log --oneline -3 -- public/openapi.json
a41f2b8 chore: regenerate openapi
8c9e114 chore: regenerate openapi
3e11fa4 chore: regenerate openapi
$ git show a41f2b8 -- public/openapi.json | head -20
- "type": "string"
+ "type": ["string", "null"]
# a field became nullable. the document was regenerated,
# the pull request was approved, and three clients found
# out in production.The generated document is a faithful description of a breaking change, produced automatically and reviewed by nobody — which is worse than no document, because it creates the impression that the contract is managed.
Why it happens
Generating from code makes the document a derivative of the implementation, so the implementation is the contract and the document is a report. Every change is automatically consistent, including the ones that break clients.
Inverting the relationship — writing the document first and asserting the implementation against it — makes a breaking change a deliberate edit to a reviewed file. That is the entire idea, and everything else is mechanism.
The fix
What 3.1 changes
3.0 3.1 (February 2021)
---------------------------------------------------------
a JSON Schema dialect with full JSON Schema 2020-12.
subtle incompatibilities the same schemas your
validators already use.
nullable: true type: ["string", "null"]
no webhooks webhooks as a top-level object
exclusiveMinimum: boolean exclusiveMinimum: number
example examples (an array)
and the tooling in Feb 2021 mostly does NOT support 3.1.The tooling gap is the practical constraint: writing 3.1 in February 2021 means the generators, the UI and the validators are at various stages of catching up, and several will reject the document outright. Staying on 3.0 for another year and designing as if 3.1 were available is the pragmatic choice.
The nullable to type-array change is worth internalising anyway because it is the one that appears in every diff, and it is the case where the two versions disagree about the same document.
The document as the input
# spec/openapi.yaml — hand-written, reviewed, and the
# thing a breaking change has to be made to
paths:
/orders/{id}:
get:
operationId: showOrder
parameters:
- name: id
in: path
required: true
schema: { type: integer, minimum: 1 }
responses:
'200':
content:
application/json:
schema: { $ref: '#/components/schemas/Order' }
'404':
content:
application/problem+json:
schema: { $ref: '#/components/schemas/Problem' }Writing it by hand is the part people resist and is the point — the friction is what makes a change to the contract a decision. A three-hundred-endpoint API is a large file and it splits into per-resource files with references, which every tool supports and which makes the review diff readable.
Asserting the implementation against it
trait AssertsAgainstSpec
{
protected function validator(): ResponseValidator
{
return (new ValidatorBuilder())
->fromYamlFile(base_path('spec/openapi.yaml'))
->getResponseValidator();
}
protected function assertMatchesSpec(
TestResponse $response,
string $operationId,
): void {
$this->validator()->validate(
new OperationAddress($operationId, 'get'),
$this->toPsr($response),
);
}
}Validation in the test suite rather than in production is deliberate: a middleware validating every response is a real cost on every request and turns a schema mistake into a 500 for a customer. In tests it is free and the failure lands on the person who caused it.
public function testShowOrderMatchesTheSpec(): void
{
$order = Order::factory()->create();
$response = $this->getJson("/api/orders/{$order->id}");
$response->assertOk();
$this->assertMatchesSpec($response, 'showOrder');
}
// and the one that is easy to forget: the error path
public function testMissingOrderMatchesTheProblemSchema(): void
{
$this->getJson('/api/orders/999999')->assertNotFound();
$this->assertMatchesSpec($response, 'showOrder', 404);
}Error responses are where the drift is worst, because they are written once and never looked at again — the 404 body from a framework default is almost never the shape the document claims. Asserting them is what makes the document trustworthy for the cases clients hit when something goes wrong.
Detecting a breaking change in the pipeline
$ npx openapi-diff spec/openapi.yaml origin/main:spec/openapi.yaml
BREAKING CHANGES
/orders/{id} GET 200 application/json
- property 'shipped_at' changed from 'string' to
'string,null'
NON-BREAKING
/orders POST 201
+ property 'reference' added
$ echo $?
1 # the build fails, and a human decidesFailing the build on a breaking change and requiring an explicit override is the mechanism that would have prevented the incident. The override is a label on the pull request rather than a configuration change, so the decision is visible in the history.
Classifying nullability as breaking is correct and is the case people argue about, since a client that handled the field as always-present now receives null. Whether that breaks them depends on their code, which is exactly why it needs a human rather than a rule.
Verifying it worked
$ vendor/bin/phpunit --testsuite=contract
Tests: 118 passed # one per operation, plus errors
$ npx openapi-diff spec/openapi.yaml origin/main:spec/openapi.yaml
No breaking changes.
$ npx spectral lint spec/openapi.yaml
✓ No results with a severity of 'error' found
# and the deliberate check, run once:
# change a field to nullable, push, watch the build fail.Deliberately introducing a breaking change to watch the pipeline reject it is the only way to know the guard works, and it is worth doing at the moment it is set up rather than trusting it. The linter is a separate concern and catches the things a validator does not — missing descriptions, inconsistent naming, operations without an id.
What this costs
The document has to be edited before the code, every time, which is a genuine change in workflow and the thing people push back on. It is also the entire benefit: a contract that can be changed as a side effect of an implementation change is not a contract. Framing it as “the API design step” rather than “documentation” is what makes it acceptable, because it is accurate.
The contract tests are a second suite that has to be maintained and they will occasionally fail for uninteresting reasons — a schema too strict about a format, a validator disagreeing with a library about a date. Each of those costs half an hour and produces a slightly better document. The ones that fail for interesting reasons pay for all of them.