Türker Yıldırım

Testing what happens when the third party is down

Deep dive10 min read

The shipping rate integration had ninety-four per cent line coverage and fell over the first time the provider returned an HTML error page instead of JSON. Every test in the suite mocked a well-formed 200 response, because that is what the documentation shows and what the sandbox returns.

The symptom

$ tail -3 /var/log/app/error.log
JsonException: Syntax error in ShippingClient.php:88
  #0 json_decode('<html><head><title>502 Bad Gateway...')

# 412 checkout failures in eleven minutes, all 500s,
# because their load balancer returned HTML and the
# client called json_decode on it unconditionally.

$ grep -c 'Http::response' tests/
41        # every one of them a 2xx with valid JSON

A parse error inside the HTTP client became a 500 at the checkout, which is our outage caused by their degradation. The client had no path for a response that was not JSON because no test had ever produced one.

Why it happens

A mock encodes what the author believes the provider returns, and the belief comes from the documentation. Documentation describes the API and not the infrastructure in front of it — the load balancer, the WAF, the rate limiter and the maintenance page are all things that answer on the same URL and are documented nowhere.

The fix

The failure modes worth covering

per integration, five tests. not more, and not fewer:

  1  connection refused / DNS failure
  2  a timeout — connect, and separately, read
  3  a 5xx with a non-JSON body (their proxy)
  4  a 200 with an error payload (success at HTTP,
     failure in the envelope — the most common one)
  5  a truncated or malformed body

and each must produce a DEFINED behaviour, not an
unhandled exception. writing the test is how you find
out there is no defined behaviour.

Case four is the one that appears in nearly every provider and is missed most often: a 200 response whose body says the operation failed. Code branching on the status code alone treats it as success and proceeds with a null.

/** @dataProvider hostileResponses */
public function testShippingFailuresDegradeToFlatRate(
    callable $fake,
    string $expectedKind,
): void {
    Http::fake(['rates.example/*' => $fake]);

    $quote = $this->shipping->quote($this->basket());

    $this->assertTrue($quote->isFallback());
    $this->assertSame($expectedKind, $quote->failureKind());
}

public function hostileResponses(): array
{
    return [
        'connection' => [fn () => throw new ConnectionException('x'), 'connection'],
        'proxy html' => [Http::response('<html>502</html>', 502), 'upstream_5xx'],
        'ok but error' => [Http::response(['error' => 'bad postcode'], 200), 'rejected'],
        'truncated'  => [Http::response('{"rates": [', 200), 'malformed'],
    ];
}

Asserting on a classified failure kind rather than on an exception type is what makes the behaviour testable at the boundary the application cares about. The caller needs to know whether to retry, to fall back or to reject the basket, and that is a smaller vocabulary than the set of things that can go wrong.

Recording real responses, including the ugly ones

// a middleware in the sandbox client, writing every
// response to a fixture directory verbatim
$stack->push(Middleware::tap(null, function ($req, $opts, $promise) {
    $promise->then(function (ResponseInterface $res) use ($req) {
        $name = $res->getStatusCode() . '-'
              . Str::slug($req->getUri()->getPath());

        file_put_contents("tests/fixtures/shipping/{$name}.http",
            Message::toString($res));      // headers AND body
    });
}));

Storing the full HTTP message rather than the decoded body is what preserves the headers, the content type and the exact bytes — and the content type is frequently the only thing distinguishing a JSON error from an HTML one. A fixture that is a PHP array has already thrown away the interesting part.

$ ls tests/fixtures/shipping/
200-v2-rates.http
200-v2-rates-empty.http
400-v2-rates.http
429-v2-rates.http          # with a Retry-After: 30
502-v2-rates.http          # text/html, from their proxy
503-v2-rates.http          # a maintenance page, 4 KB of HTML

# five of these six would never have been written by hand.

A timeout that is a timeout

// the client, with both timeouts set explicitly
$response = Http::timeout(4)          // total, including body
    ->connectTimeout(2)
    ->retry(2, 200, throw: false)
    ->get($url);

// and the test for a SLOW response rather than a hung one:
// a body that trickles is not covered by connectTimeout
Http::fake(['rates.example/*' => function () {
    usleep(5_000_000);

    return Http::response(['rates' => []], 200);
}]);

A connect timeout without a total timeout means a server that accepts the connection and then sends one byte per second holds the request forever, which is the failure that takes down a worker pool rather than one request. Setting both explicitly is two lines and the default for one of them is usually infinite.

Simulating the slow case in a test is awkward — a real sleep makes the suite slow — and it is worth one test that takes five seconds to prove the timeout fires. Marking it as a separate group so it does not run on every push is the compromise.

A contract test against their document

// nightly, against the sandbox — not on every push
public function testSandboxStillMatchesTheirSpec(): void
{
    $validator = (new ValidatorBuilder())
        ->fromJsonFile(base_path('specs/shipping-openapi.json'))
        ->getResponseValidator();

    $response = $this->liveClient->post('/v2/rates', $this->sample());

    $validator->validate(
        new OperationAddress('/v2/rates', 'post'),
        $response,
    );
}

This detects their change rather than ours, so it belongs on a schedule and in a channel rather than blocking a deploy. It is the only thing that will tell you a field became nullable before a customer does, and it only works if the provider publishes a machine-readable specification — which in 2021 many still do not.

Verifying it worked

$ vendor/bin/phpunit --filter Shipping
Tests: 23, Assertions: 61

# and the deliberate outage, on staging
$ iptables -A OUTPUT -d rates.example -j REJECT
$ curl -s /api/checkout/quote | jq -r '.shipping.source'
"flat_rate_fallback"
$ curl -s -o /dev/null -w '%{http_code}n' /api/checkout/quote
200

$ iptables -D OUTPUT -d rates.example -j REJECT

Blocking the provider at the firewall on staging and asserting that checkout still works is the acceptance test, and it exercises the whole path rather than the client class. It is worth running as part of a game day rather than in CI, because it needs a real environment and a person watching.

A 200 with a fallback rate rather than a 500 is the difference this bought: the customer gets a shipping price that is slightly wrong instead of an error page, and somebody gets an alert. Whether a wrong price is acceptable is a business decision that had to be asked rather than assumed.

What this costs

Fixtures that go stale, because a recorded response is a snapshot of an API that changes. The nightly contract test is the mechanism that detects the staleness and it needs somebody to act on it — a failing scheduled job that nobody reads is worse than none, because it creates the impression of coverage.

The fallback behaviour is also a decision that has to be revisited. A flat rate that was reasonable in 2021 is wrong by 2023, and nothing about the code will say so. Putting the fallback values in configuration with a review date attached is the least bad answer, and it is a review that will be skipped.