A circuit breaker before the third-party outage

The payment provider never went down. It got slow — responses moved from 300 milliseconds to about twenty seconds — and within four minutes the entire site was returning 502, including pages that have nothing to do with payment. The provider recovered on its own; we did not, because by then the queue of waiting requests was longer than the outage.

The symptom

$ tail -f /var/log/php-fpm/error.log
[10-Dec-2017 14:41:02] WARNING: server reached pm.max_children setting (50)
[10-Dec-2017 14:41:03] WARNING: server reached pm.max_children setting (50)

$ curl -s localhost/status?full | grep -c 'state: Running'
50

$ curl -s localhost/status?full | grep 'request uri' | sort | uniq -c
     47 request uri: /checkout/pay
      3 request uri: /

# 47 of 50 workers waiting on one upstream. the other 3 are the site.

The homepage was down because there were three workers left to serve it. Nothing about the homepage had changed, and nothing about it was slow — it simply could not get a worker.

Why it happens

The call had a thirty-second timeout, which felt responsible. A timeout bounds how long a single request waits; it says nothing about how many can wait simultaneously. With fifty workers and a twenty-second upstream, the pool is exhausted at two and a half requests per second — a rate the site exceeds at lunchtime.

The arithmetic is worth writing down because it is the entire argument: workers ÷ upstream latency = maximum request rate. A dependency getting slower does not degrade a system proportionally. It reduces capacity, and capacity falling below demand is a cliff rather than a slope.

The fix

Three states, and what each one does

A circuit breaker tracks failures against a dependency and stops calling it when there have been too many. Closed passes calls through. Open rejects them immediately without a network call. Half-open lets exactly one through to find out whether recovery has happened.

public function call(callable $operation, callable $fallback)
{
    if ($this->isOpen() && ! $this->shouldAttemptReset()) {
        return $fallback();              // no network call at all
    }

    try {
        $result = $operation();
        $this->recordSuccess();

        return $result;
    } catch (RequestException $e) {
        // a 402 is the provider working correctly. do not count it.
        if ($e->getResponse() && $e->getResponse()->getStatusCode() < 500) {
            throw $e;
        }

        $this->recordFailure();

        return $fallback();
    }
}

The distinction in that second catch is what separates a useful breaker from one that trips during normal business. A declined card is a 402 and is the provider doing its job; counting it as a failure means a run of declines opens the circuit and stops payments entirely.

Shared state, because workers are separate processes

Counting failures in a PHP object means each of fifty workers keeps its own count and each opens independently after its own threshold — fifty times more slow calls than intended. The state has to live where every worker can see it.

private function recordFailure()
{
    $key = "cb:{$this->service}:failures";

    // INCR is atomic; the expiry makes it a rolling window
    $count = $this->redis->incr($key);

    if ($count === 1) {
        $this->redis->expire($key, $this->windowSeconds);
    }

    if ($count >= $this->failureThreshold) {
        $this->redis->setex("cb:{$this->service}:open", $this->cooldown, 1);

        Log::warning('circuit.opened', [
            'service'  => $this->service,
            'failures' => $count,
        ]);
    }
}

Using an expiring key as the open state removes the need for a timer or a scheduled reset: the circuit closes when the key expires, and half-open is simply the first call after it has. That is a good deal less code than most descriptions of the pattern suggest.

Choosing the thresholds, which is the part nobody writes about

Descriptions of this pattern say “five failures” as though the number were obvious. It is derived from the capacity arithmetic, not chosen.

// 50 workers, no more than 20% ever waiting on one dependency
// = 10 concurrent slow calls tolerated, so 10 failures per 30s window

'payments' => [
    'timeout'           => 3,     // the single most valuable change
    'failure_threshold' => 10,
    'window_seconds'    => 30,
    'cooldown_seconds'  => 30,
],

// a legitimately slow dependency gets its own budget
'reporting' => ['timeout' => 20, 'failure_threshold' => 3],

Dropping the timeout from thirty seconds to three did more than the breaker itself. Thirty seconds was never a real number — no user waits that long, and the only thing it bought was a worker held hostage for half a minute.

The fallback, which is decided per call

There is no generic answer to what happens when the circuit is open, and a global default is how a breaker causes silent data loss. Each call site needs its own.

// payment authorisation — no fallback exists. fail honestly, 503.
$breaker->call($authorise, function () { throw new PaymentUnavailable(); });

// address lookup — degrade to manual entry; the form still works
$breaker->call($lookup, function () { return []; });

// analytics — queue it and move on
$breaker->call($send, function () use ($event) {
    return Queue::later(60, new SendEvent($event));
});

Failing honestly is a legitimate fallback and often the right one. The improvement over the original incident is not that payment keeps working — it cannot — but that it fails in eight milliseconds and the rest of the site stays up.

Verifying it worked

# take the provider away deliberately, on staging
$ iptables -A OUTPUT -d 203.0.113.40 -j DROP

$ ab -n 500 -c 20 https://staging.shop/checkout/pay
Requests per second:    2140.11 [#/sec]
Time per request:       9.3 [ms]
Non-2xx responses:      500

$ curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://staging.shop/
200 0.081                      # the site is up

$ redis-cli get cb:payments:open
"1"
$ redis-cli ttl cb:payments:open
(integer) 22

Five hundred failures in a quarter of a second, and the homepage unaffected. Before the change the same test exhausted the pool in four seconds and took the whole site with it — which is the comparison worth keeping in the runbook, because it is what justifies the added complexity to whoever inherits this.

Restoring the rule and watching the circuit close on its own is the other half of the test. A breaker that opens correctly and never recovers is a worse failure than no breaker, and it is entirely possible to write one by mistake.

What this costs

A breaker that trips wrongly is an outage you caused, and the ways to trip wrongly are not exotic: counting 4xx responses as failures, a threshold set below normal error volume, or a cooldown long enough that a two-second blip becomes a minute of rejected payments. Every one of those is a configuration mistake rather than a code bug, which means it will not be caught by tests and will be discovered in production.

The state also has to be visible or nobody will trust it. A circuit that is open with no dashboard and no alert produces the worst kind of incident — one where the third-party dependency has recovered, the application is still refusing to call it, and the person debugging has no idea the breaker exists. Logging every transition and alerting on a circuit that stays open beyond a couple of cooldowns is not optional; it is the difference between a safety mechanism and a trap.