An API that had two clients and three shapes

One endpoint returning an order, and three different response shapes depending on who asked. One of them was designed; the other two had accumulated one conditional at a time, each added for a consumer that needed something slightly different.

The symptom

public function toArray($request): array
{
    $data = [ /* the twenty-one fields */ ];

    if ($request->header('X-Client') === 'mobile') {
        unset($data['description'], $data['metadata']);
    }

    if ($request->query('expand') === 'full') {
        $data['lines'] = LineResource::collection($this->lines);
    }

    if ($request->user()?->apiKeyId === 'legacy-integrator') {
        $data['total'] = $this->total->cents / 100;   // a float
    }

    return $data;
}

The third conditional is the one to look at: one named consumer receives money as a float because that is what they parsed in 2019. The API has a shape and a set of exceptions, and the exceptions are keyed on identifiers rather than on anything a caller could discover.

Why it happens

A field added for one consumer, conditionally, is the smallest possible change and it is always the right thing to do in the moment. Three of them is a response shape nobody can describe, and the specification describes none of them.

The fix

Finding the shapes rather than guessing them

// a fortnight of logging what was actually returned
Log::channel('shapes')->info('order.show', [
    'api_key' => $request->user()?->apiKeyId,
    'keys'    => array_keys($data),
    'expand'  => $request->query('expand'),
]);

two weeks, grouped by the sorted key list:

  shape A  21 fields          88,204 requests   3 keys
  shape B  19 fields (mobile)  41,208           1 key
  shape C  22 fields (+lines)   4,102           2 keys
  shape D  21 fields, total
           as a float             902           1 key

four shapes, not three. and shape D's consumer had been
assumed to be gone since 2022.

Logging the key list rather than the payload is what makes this cheap and safe — no personal data, one line per request, and the grouping falls out. The fourth shape is the finding: a consumer everybody believed had migrated, still making nine hundred requests a day.

One shape, and an explicit include

public function toArray($request): array
{
    return [
        ...$this->base(),
        ...$this->when($this->includes($request, 'lines'), fn () => [
            'lines' => LineResource::collection($this->lines),
        ]),
        ...$this->when($this->includes($request, 'refunds'), fn () => [
            'refunds' => RefundResource::collection($this->refunds),
        ]),
    ];
}

private function includes(Request $r, string $name): bool
{
    return in_array($name, $r->includes(), strict: true);
}

// and the allow-list, per endpoint, published in the
// OpenAPI document
public const array ALLOWED_INCLUDES = ['lines', 'refunds', 'customer'];
public const int MAX_INCLUDE_DEPTH = 2;

// a request naming something else is a 400 with the
// permitted values, rather than a silently ignored
// parameter.

Rejecting an unknown include rather than ignoring it is the decision that makes this an API rather than a set of behaviours — a consumer who misspells a value finds out immediately instead of wondering why the field is absent. The allow-list is also what the specification is generated from, so the two cannot drift.

The consumer who depended on a field being absent

the mobile client had been receiving 19 fields and
parsing with a strict decoder that rejects unknown keys.

so moving them to one shape — 21 fields for everybody —
broke them, in a way that returning MORE data normally
does not.

which means "additive changes are always safe" has an
exception, and the exception is a strict parser.

the fix was theirs and the sequencing was ours: sparse
fieldsets shipped first, they adopted `fields=`, and
the shape unification followed six weeks later.

A strict decoder makes an additive change breaking, which is the one case that overturns the usual rule. It is worth knowing which consumers have one, and the answer is not discoverable from our side — it came from asking, which is why having four named integrators rather than an open API made this tractable.

The float, deprecated with a date

HTTP/1.1 200 OK
Deprecation: Sat, 01 Jun 2024 00:00:00 GMT
Sunset: Tue, 01 Oct 2024 00:00:00 GMT
Link: <https://docs.example.com/api/migrations/money>; rel="deprecation"

plus an email to the one named consumer, because the
headers are ignored by everybody and the email is what
actually produces a migration.

The headers are the standard mechanism and the email is the effective one, which is the honest ordering. Four months for a consumer to change a parser is generous and it is what a small integrator with no dedicated engineering time actually needs.

The cost budget

// an include has a weight; the sum is checked before
// the query runs
private const array INCLUDE_COST = [
    'lines' => 1, 'refunds' => 1, 'customer' => 1,
    'customer.addresses' => 3, 'shipments.tracking' => 5,
];

if ($this->costOf($includes) * $perPage > self::BUDGET) {
    throw new IncludeBudgetExceeded(
        'reduce per_page or the number of includes',
    );
}

A consumer requesting five includes on a two-hundred-item page is a request that will run for forty seconds and is not malicious — it is somebody reading the documentation and using what is offered. Rejecting it with an actionable message is better than serving it, and much better than a timeout.

Verifying it worked

$ ./bin/shape-report --since=30d
  distinct shapes: 1 (plus declared includes)
  requests with an unknown include: 4 (all 400s, all
    from one consumer's first attempt)

$ curl -s '/api/orders/8814?include=nonsense' | jq -c .
{"type":"...","title":"Unknown include",
 "detail":"'nonsense' is not available. Permitted:
 lines, refunds, customer."}

$ vendor/bin/phpunit --group=contract
  4 consumer fixtures, 41 assertions.  OK

$ ./bin/openapi-diff --against=published
  no undocumented response fields

A single shape in thirty days of traffic is the assertion, and the four rejected includes are the mechanism working — a consumer guessing a name, being told the permitted values, and getting it right on the second attempt.

What this costs

An include parameter that will grow until it is a query language. Three values today, a documented depth limit of two, and a cost budget — all of which are the guard rails somebody will ask to relax the first time a consumer needs a fourth level of nesting.

The six-week sequencing also depended on being able to email four people, which is not an API with public consumers. The same change on an open API would have needed a version, and the reason it did not need one here is a property of the business rather than of the design.