The API had three versions in production. v3 was current, v2 had been deprecated in 2020, and v1 had been deprecated in 2019 with a removal date that had passed twice. Every change to a shared model meant touching three sets of resources, three sets of tests and three sets of documentation, and nobody could say who was still calling v1.
The symptom
$ ls app/Http/Resources/
V1/ V2/ V3/
$ find app/Http/Resources -name '*.php' | wc -l
88 # for 31 distinct resources
$ git log --oneline --since='6 months' -- app/Http/Resources | wc -l
41
$ grep -rn 'deprecated' routes/api-v1.php | head -1
// deprecated 2019-04-01, remove 2019-10-01
# three and a half years past its removal date.Eighty-eight resource classes for thirty-one things is the cost, and the removal dates in comments are the symptom of a deprecation that had no mechanism behind it. Nobody was ignoring the dates deliberately; nothing happened when they passed.
Why it happens
Removing a version requires knowing that nobody is using it, and knowing that requires measurement that was never added. Without it the only available action is an announcement followed by hope, and the risk of breaking an unknown client is always larger than the cost of maintaining the version for another quarter.
The fix
Measuring, per version and per client
// a middleware on every versioned route
$this->metrics->increment('api_requests_total', [
'version' => $route->version(), // v1 | v2 | v3
'endpoint' => $route->getName(),
'client' => $token->clientId(), // a bounded set
]);
// client id is safe as a metric label because it is a
// small known set. a user id or an IP is not — that is a
// cardinality bomb, and it belongs in a log line.$ ./bin/api-usage --version=v1 --days=30
client requests last seen
acme-integration 412,008 2022-09-14 09:41
brightwork-sync 88,104 2022-09-14 09:38
legacy-reporting 1,204 2022-08-02 03:00
(unattributed) 41 2022-09-11 14:22
# four clients. and the fourth is the interesting one.The unattributed forty-one requests are the finding: a client authenticating as a user rather than through a registered application, which no client-id label can attribute. Tracing those took a week and produced a fifth integration nobody knew existed — built in 2020 by somebody who had left.
The sunset header, and telling clients before emailing them
// RFC 8594, on every v1 and v2 response
return $response
->header('Deprecation', 'Sat, 01 Oct 2022 00:00:00 GMT')
->header('Sunset', 'Wed, 01 Feb 2023 00:00:00 GMT')
->header('Link', '<https://docs.example/api/v3-migration>; '
. 'rel="deprecation"; type="text/html"');
// Deprecation: it is deprecated as of this date
// Sunset: it stops working on this date
// Link: where to read about itAlmost no client library surfaces these automatically, which means the header is a dated record rather than a notification — its value is that a conversation in January has something unambiguous to point at. Logging which clients receive one and do not change behaviour is what turns it into a list of who to email.
A migration path that is a diff
the documentation that was written, per endpoint:
GET /v1/orders/{id} → GET /v3/orders/{id}
renamed:
order_total → total.amount_cents
order_currency → total.currency
placed → placed_at (now ISO 8601, was a
unix timestamp)
removed:
legacy_reference → no equivalent. it was a column
dropped in 2020 and has returned
null ever since.
added:
tax → a new object
a jq expression that converts a v3 response to the v1
shape, for a client that wants a two-line adapter
rather than a rewrite.The jq expression is what turned two of the four clients from a rewrite into an afternoon, and writing it took an hour. A migration guide that lists changes without providing a mechanical translation puts the whole cost on the client, which is why deprecations stall.
The legacy_reference entry is the honest one: a field that had been returning null since 2020 and that one client was still reading. Documenting that it is gone rather than quietly removing it is the difference between a migration and a surprise.
The brownout
announced two weeks in advance, to a specific date:
2022-11-15 10:00–10:15 v1 returns 410 Gone
2022-11-22 10:00–11:00 v1 returns 410 Gone
2022-12-06 10:00–17:00 v1 returns 410 Gone
2023-02-01 v1 removed
the first brownout found:
a monitoring check nobody had attributed
a client who had migrated their main flow and left one
endpoint on v1
the fifth integration, which failed loudly and was
finally traced to a machine in a cupboard
fifteen minutes, and it produced three phone calls that
six months of emails had not.The brownout is the step people skip and it is the one that converts uncertainty into a list of names. It has to be announced, short, and at a time when the people who can respond are awake — and it works precisely because it is disruptive enough to be noticed and short enough to be tolerable.
// implemented as a feature flag with a schedule, so it is
// a configuration change rather than a deploy
if ($this->brownout->isActive('api-v1')) {
return response()->json([
'type' => 'https://docs.example/probs/version-sunset',
'title' => 'API v1 is being retired',
'status' => 410,
'detail' => 'This is a scheduled brownout. v1 will be '
. 'removed on 2023-02-01. See the Link header.',
], 410)->header('Link', '<https://docs.example/api/v3-migration>; '
. 'rel="deprecation"');
}Returning a body that explains what is happening is what makes the brownout a message rather than an outage, and 410 rather than 404 is deliberate — it says the resource is gone permanently rather than that the URL is wrong. A client debugging a 404 looks for a typo.
The four clients, resolved
acme-integration migrated. two weeks, with the jq
adapter as a bridge for one endpoint.
brightwork-sync migrated. one afternoon.
legacy-reporting a nightly job at a client who had
been acquired. no engineering team.
→ we wrote the adapter, they deployed
it. four hours of our time, billed.
the fifth a machine in a cupboard running a
2020 script. switched off, with the
owner's agreement, after the second
brownout proved nobody noticed.
and the account manager was in every one of these
conversations, which is why they went well.Writing the adapter for a client with no engineering team is a commercial decision rather than a technical one, and it was cheaper than another year of maintaining v1. Having the account manager involved from the first email is what made that option available — an engineering-only deprecation has no mechanism for “we will do it for you”.
What made three versions expensive, specifically
The cost of a version is not the resource classes — those are cheap to duplicate. It is that every shared change has to be reasoned about three times, and the reasoning is where the mistakes are.
// a field added to the model. what happens in each version?
// v3: added to the resource. straightforward.
'tax' => new TaxResource($this->tax),
// v2: added? it is additive, so it is "safe" — unless a
// client validates strictly, which one of them does.
// so: not added, and v2 diverges further from the model.
// v1: definitely not. and now the v1 resource references
// a relationship that the v3 query eager-loads and the v1
// controller does not, which is an N+1 nobody notices
// because v1 has 400 requests a day.The N+1 in a low-traffic version is the shape of every bug in an old API version: invisible in the metrics because the traffic is small, and discovered when a client asks why one endpoint takes four seconds. Three of the four bugs found in v1 over its lifetime were of that kind.
and the three-times cost, measured over six months:
changes touching all three versions 23
average extra time per change ~40 minutes
→ about 15 hours
bugs found in v1 or v2 only 4
→ all four caused by a change applied to v3 correctly
and to the others carelessly
test suite time attributable to v1/v2 3m 20s of 12m
fifteen hours and four bugs is the number that justified
the removal. it is not large, and it recurs every year.Fifteen hours over six months is a small number and it is the honest one — the case for removal is that it recurs indefinitely and grows with every version, not that it is dramatic today. Presenting it as a large saving would have been quoted back when the removal took three months of coordination.
What the removal actually removed
$ find app/Http/Resources -name '*.php' | wc -l
62 # was 88
$ find tests -path '*V1*' -name '*.php' -delete
$ vendor/bin/phpunit
Tests: 1,188 passed # was 1,412
$ cloc app/Http --quiet | grep PHP
PHP 188 files 4,102 blank 2,844 comment 18,204 code
# 6,102 lines removed
# and the change that is not measurable: every future
# change to a shared model is now made twice, not three
# times.Six thousand lines and two hundred and twenty-four tests removed is the visible outcome, and the ongoing saving is the one that matters — a third of the cost of every future API change, indefinitely. That is the number to quote when the next deprecation is proposed and somebody asks whether it is worth it.
Verifying it worked
$ ./bin/api-usage --version=v1 --days=30
no requests
$ curl -si https://api.example/v1/orders/1 | head -1
HTTP/2 410
$ ./bin/api-usage --version=v2 --days=30
client requests last seen
acme-integration 88,104 2022-09-14
# v2 is next, and the same process applies
# thirty days at zero before removal, which was the
# criterion agreed in advance.Thirty days at zero as a written criterion is what made the removal a scheduled task rather than a judgement call, and agreeing it before the brownouts meant nobody had to decide under pressure. v2 entered the same process the following month with a much shorter timeline, because the mechanism now existed.
What this costs
A conversation with somebody’s account manager, and in one case four hours of engineering time given away to a client who could not do the migration themselves. That is a commercial cost that engineering cannot authorise, which is why the deprecation had stalled for three years — the mechanism was missing, and the mechanism is organisational rather than technical.
The brownouts are also genuinely disruptive and will produce complaints, including from clients who were told twice and did nothing. Being able to point at a dated header in their own logs, an email with a read receipt and two announced windows is what makes that conversation short — and every one of those artefacts has to exist before the first brownout rather than after the first complaint.
It is also worth saying that the measurement should have existed from the day v1 shipped. Adding it retrospectively took two days and would have taken twenty minutes at the start, and every deprecation since has been a scheduled operation rather than a three-year stalemate.