The payment gateway had a forty-second outage at 11:04. Our systems were degraded until 11:52, which is forty-eight minutes of our own doing — every client retried at the same moment the gateway came back, and knocked it over again, four times.
The symptom
11:04:12 gateway starts returning 503
11:04:52 gateway recovers
11:04:53 4,102 queued retries fire simultaneously
11:04:56 gateway returns 503 again
11:05:56 gateway recovers
11:05:57 8,204 queued retries fire simultaneously
11:05:59 gateway returns 503 again
...
11:52:04 the queue is drained by hand and it stabilises
the original outage: 40 seconds.
ours: 48 minutes.Each recovery was met with a larger burst than the one before, because the retry backlog accumulated during the failure. The gateway never got a chance to warm up before being hit with more traffic than it sees at peak.
Why it happens
Exponential backoff spreads one client’s retries over time and does nothing about a thousand clients whose clocks are aligned by the same failure. Every one of them failed at 11:04:12 and every one of them retried at 11:04:12 plus one second, then plus two, then plus four.
The backoff curve is doing exactly what it was designed to do and the synchronisation defeats it. This is not a bug in anybody’s retry implementation; it is a property of a shared trigger.
The fix
Jitter, which matters more than the curve
// synchronised: every client retries at 1s, 2s, 4s, 8s
$delay = 2 ** $attempt;
// full jitter: uniformly random in [0, window]
$delay = random_int(0, (2 ** $attempt) * 1000) / 1000;
// decorrelated jitter: converges faster, bounded by a cap
$delay = min($cap, random_int($base, (int) ($previous * 3)));simulated: 4,000 clients, a 40-second outage, 5 attempts
no jitter peak 4,000 req/s at t+41
equal jitter peak 1,204 req/s at t+41
full jitter peak 188 req/s, spread over 32s
decorrelated peak 204 req/s, drained soonest
full jitter halves the average delay and cuts the peak
by a factor of twenty. that trade is almost always right.Full jitter is the version with the best published behaviour and the one people find least intuitive, because randomising down to zero feels like it should be worse. The peak is what matters during a recovery, not the average, and the peak is what full jitter destroys.
A circuit breaker with a half-open state
// closed → requests pass, failures counted
// open → requests fail immediately, no call made
// half-open → ONE request passes. success closes, failure
// reopens the timer.
public function call(callable $fn): mixed
{
return match ($this->state()) {
State::Closed => $this->callAndCount($fn),
State::HalfOpen => $this->probe($fn),
State::Open => throw new CircuitOpen($this->retryAfter()),
};
}The half-open state is what makes the recovery decision evidence-based rather than a timer, and it must admit exactly one request. Several workers each sending a probe is the thundering herd the breaker exists to prevent, recreated inside the breaker.
private function probe(callable $fn): mixed
{
// one probe, cluster-wide. everything else fails fast.
$lock = Cache::lock("breaker:{$this->name}:probe", 10);
if (! $lock->get()) {
throw new CircuitOpen($this->retryAfter());
}
try {
$result = $fn();
$this->close();
return $result;
} catch (Throwable $e) {
$this->open();
throw $e;
} finally {
$lock->release();
}
}The distributed lock is not optional in any multi-process deployment, and it is the part that is missing from most implementations copied from a blog post. Without it, six workers in half-open state send six probes, which for a service that fell over at four thousand requests is a smaller herd and the same mechanism.
Thresholds that are a rate, not a count
// a consecutive counter never trips on a service failing
// one request in three, which is unambiguously unhealthy
if (++$this->consecutiveFailures >= 5) {
$this->open();
}
// a rate over a rolling window does
$window = $this->window(60);
if ($window->total() >= 20 && $window->failureRate() > 0.5) {
$this->open();
}The minimum volume in the condition is what stops the breaker opening on two failures out of two during a quiet period, which is a real and irritating failure mode. Twenty requests in sixty seconds is a threshold that needs to be derived from the actual traffic rather than copied.
A retry budget, so the total load is bounded
// per-caller limits multiply: 3 attempts each × 4,000
// clients = 12,000 requests at exactly the wrong moment
// a budget is a shared counter, and it self-disables
final class RetryBudget
{
public function permit(): bool
{
$successes = $this->window->successesLastMinute();
$retries = $this->window->retriesLastMinute();
// retries may be at most 10% of the success rate
return $retries < max(10, $successes * 0.1);
}
}The property that matters is that it self-disables: during an outage there are almost no successes, so there is almost no budget, so retries stop entirely. A per-caller limit does the exact opposite and multiplies the load precisely when the dependency is least able to absorb it.
The floor of ten permits some retrying even from a standing start, which is necessary — a budget that reaches zero and stays there never recovers on its own. Choosing that floor is the one arbitrary number in the design.
Which errors are retryable
retryable:
connection refused, DNS failure, connect timeout
502, 503, 504
429, but ONLY after the Retry-After it gave you
a read timeout on an IDEMPOTENT request
never retryable:
400, 401, 403, 404, 422 — the request is wrong, and
it will be wrong again
409, usually — a conflict needs a decision
a read timeout on a NON-idempotent request, unless
there is an idempotency key
the last one is the expensive mistake: the operation may
have succeeded, and retrying charges the card twice.Retrying a timeout on a non-idempotent operation is the error that costs money rather than availability, and it is the default behaviour of most HTTP client retry helpers. The idempotency key is what makes it safe, and without one the correct behaviour is to fail and let a human or a reconciliation job resolve it.
The queue that was holding the backlog
None of the above explains why the second burst was larger than the first. The retries were not being made by four thousand independent callers — they were jobs on a queue, and a failed job goes back onto it.
// the default: release back to the queue, immediately
public function handle(): void
{
$this->gateway->capture($this->order); // throws
}
// what actually happens on failure:
// the job is released with a delay of 0
// the worker picks it up again within milliseconds
// it fails again, and the attempt counter climbs
// → a hot loop against a service that is already down
// the backoff must be on the JOB, not only in the client
public function backoff(): array
{
return [
random_int(5, 15),
random_int(30, 90),
random_int(120, 360),
];
}A job that fails and is released with no delay retries as fast as the worker loop allows, which is several times a second per worker. That is a tighter loop than any HTTP client retry policy and it is invisible from the client’s configuration — the backoff in the HTTP client was carefully jittered and the queue underneath it was not.
Returning an array from backoff gives a different delay per attempt, and randomising each one is what spreads the release times across the pool. Without it, four thousand jobs released at the same instant with the same delay are back in lockstep after one attempt regardless of what the client does.
// and the part that stops the queue itself amplifying:
// when the breaker is open, do not consume at all
public function handle(): void
{
if ($this->breaker->isOpen()) {
$this->release($this->breaker->retryAfter());
return; // no call made, no failure recorded
}
$this->breaker->call(fn () => $this->gateway->capture($this->order));
}Checking the breaker before doing the work turns a queue of four thousand failing jobs into a queue of four thousand jobs that are cheaply deferred, which costs one Redis read each rather than one connection attempt each. That change alone removed most of the load on the recovering gateway.
It also means the attempt counter stops climbing during an outage, which matters because otherwise a forty-second failure exhausts the retry budget of every job in flight and sends them all to the failed table. That was the other half of the forty-eight minutes: several hundred jobs had to be replayed by hand afterwards.
Timeouts that decrease inward
client 30s
gateway 25s
api 20s
db query 5s ← room for two retries
gateway API 8s ← and a fallback
if the inner call can take as long as the outer request
is allowed, there is no time left to retry it or to
return a useful error — the caller has already gone and
the work continues, unobserved.The equal-timeout failure wastes resources invisibly: the client has disconnected, and the request is still running, holding a worker and a connection. A budget passed down the call chain is the rigorous version; a documented ladder of constants gets most of the benefit and needs somebody to maintain it.
Verifying it worked
# fault injection: drop the dependency, restore it, measure
$ ./bin/chaos --target=gateway --duration=40s
t+00 gateway rejected
t+02 breaker OPEN (failure rate 0.94 over 41 requests)
t+02 requests fail fast, 1.2ms, with a fallback
t+32 half-open probe #1 → fail, reopen
t+40 gateway restored
t+62 half-open probe #2 → success, breaker CLOSED
t+63 peak 204 req/s, drained by t+71
our degradation: 63s, against a 40s outage.
previously: 48 minutes.Sixty-three seconds against a forty-second outage is the number, and the twenty-three seconds of overhang is the breaker’s probe interval — which is a deliberate trade rather than a fault. Running this as a scheduled fault injection is what keeps it true after the next refactor.
The peak of two hundred requests per second on recovery, against four thousand before, is what stopped the cascade. That single number is the whole result of the jitter change.
What this costs
A failure path that is now three interacting mechanisms — jitter, a breaker with a distributed lock, and a shared budget — each with its own state and its own thresholds. Debugging why a request failed fast now means knowing which one refused it, which is why every rejection has to log a reason rather than an exception type.
The thresholds are also numbers that were correct at the moment they were measured. Traffic doubles, the minimum volume becomes too low, the breaker opens on noise, and somebody disables it during an incident — which is the most likely way this arrangement fails. Reviewing them alongside the capacity plan is the mitigation and it is a meeting nobody schedules.
It is also worth saying that none of this made the gateway more reliable. It made our response to their unreliability proportionate, which is a different and smaller claim than the architecture diagram suggests.