The circuit breaker that tripped on its own health check

On a Thursday afternoon in mid-September, our checkout service tripped its circuit breaker to the upstream payment provider. For forty-two minutes, every customer payment was rejected with an immediate 503 error before reaching the gateway. The payment provider had zero downtime, their status page was clean, and our synthetic network pings to their API resolved in under twenty milliseconds. The circuit breaker had tripped on its own probe traffic.

The symptom

14:18:02  Traffic spikes by 35% following a marketing broadcast.
14:18:14  Worker pool saturation reaches 94%.
14:18:20  Synthetic health probe to /health/upstream times out (timeout=2000ms).
14:18:25  Consecutive probe timeouts: 5 of 5.
14:18:26  Circuit state transitions CLOSED -> OPEN.
14:18:26  Fast-fail kicks in: 100% of real checkout requests rejected locally.
14:22:00  Half-open canary probe scheduled. Canary probe times out in worker queue.
14:22:01  Circuit resets back to OPEN.

The paradox of an automated protection mechanism: the upstream service was healthy, but our own worker pool was too congested to execute the health probe before the two-second timeout elapsed. The breaker interpreted local thread contention as upstream failure, locked the door, and worsened the traffic backlog.

Why it happens

The implementation made three classical assumptions. First, it shared the underlying HTTP transport pool between health checks and customer requests. Second, it evaluated probe failure inside the same rolling error-rate sliding window as real transactions. Third, the probe was dispatched synchronously by an incoming web request whenever the half-open timer expired, which meant a queued customer request bore the latency penalty of testing a potentially dead external system.

The fix

Bulkheading the probe transport

final class PaymentGatewayCircuitBreaker
{
    private ClientInterface $customerClient;
    private ClientInterface $probeClient;
    private RedisCircuitStorage $storage;

    public function __construct(
        ClientInterface $customerClient,
        ClientInterface $probeClient,
        RedisCircuitStorage $storage
    ) {
        $this->customerClient = $customerClient;
        $this->probeClient = $probeClient;
        $this->storage = $storage;
    }

    public function execute(callable $action): mixed
    {
        $state = $this->storage->getState('payment_gateway');

        if ($state === CircuitState::Open) {
            throw new CircuitOpenException('Payment gateway currently unavailable.');
        }

        try {
            $result = $action($this->customerClient);
            $this->storage->recordSuccess('payment_gateway');
            return $result;
        } catch (UpstreamConnectionException $e) {
            $this->storage->recordFailure('payment_gateway');
            throw $e;
        }
    }
}

The probe mechanism was entirely decoupled from customer traffic. Instead of passive in-band detection during user requests, an out-of-process daemon (a dedicated systemd timer) executes the canary ping using a separate HTTP connection pool. If user workers are saturated, the canary continues to execute independently with dedicated CPU slices.

Verifying it worked

# Simulate 100% PHP-FPM pool saturation on staging:
$ wrk -t8 -c250 -d30s http://staging.internal/compute-heavy

# While workers are saturated, simulate upstream latency via traffic control:
$ sudo tc qdisc add dev eth0 root netem delay 2500ms

# Probe daemon output:
[14:40:02] Probe failed (2504ms). Consecutive failures: 1/5.
[14:40:07] Probe failed (2501ms). Consecutive failures: 2/5.
# Circuit remains CLOSED until threshold.

$ sudo tc qdisc del dev eth0 root
[14:40:12] Probe succeeded (18ms). Counter reset to 0. Circuit healthy.

Simulating pool saturation no longer triggers false circuit openings. The probe metrics reflect actual network latency to the gateway rather than waiting time in our local socket queues.

What this costs

An extra background process and Redis state synchronization. The breaker state can no longer live in APCu on the local web worker; it must be stored in Redis so the out-of-process prober and the application workers agree on the state within a hundred milliseconds.

And threshold calibration. Setting the error window too high delays failure detection; setting it too low makes it sensitive to transient packet drops. A circuit breaker is not an install-and-forget component; it is an operational dependency that requires runbook drills every quarter.