The webhook we sent, and the retries we owed

The API had consumed webhooks from payment providers for years and had never sent any. In December it started sending order events to integrators, and the first outage on a subscriber’s side revealed that every question a subscriber might ask — did you send it, will you retry, when do you give up — had no answer.

The symptom

the first support ticket, on day four:
  "our endpoint was down 09:00-15:00 yesterday. which
   events did we miss, and will you resend them?"

and what could be answered:
  which events  no. there was no delivery record, only a
                job that had run and failed.
  will you      the queue retried 3 times over 10 minutes
  resend        and dead-lettered. so: no.
  can you now   only by re-triggering the source events.

six hours of events, lost, on day four.

A retry policy inherited from the queue defaults is three attempts over ten minutes, which is correct for an internal job and is not a delivery guarantee to a third party. Nothing about the implementation was wrong; it had simply been built as a job rather than as a product.

Why it happens

Sending a webhook looks like dispatching a job, so it is built as one — and a job has a retry policy chosen for internal failures, no record a third party can see, and no notion of a subscriber whose endpoint is permanently gone.

The fix

A delivery is a record, not a job

CREATE TABLE webhook_deliveries (
  id            BINARY(16) PRIMARY KEY,
  endpoint_id   BIGINT UNSIGNED NOT NULL,
  event_id      BINARY(16) NOT NULL,
  payload       JSON NOT NULL,
  attempts      TINYINT UNSIGNED NOT NULL DEFAULT 0,
  state         ENUM('pending','delivered','failed','abandoned') NOT NULL,
  last_status   SMALLINT UNSIGNED NULL,
  last_body     VARCHAR(512) NULL,      -- an excerpt
  next_attempt_at DATETIME(6) NULL,
  created_at    DATETIME(6) NOT NULL,

  KEY idx_due (state, next_attempt_at)
) ENGINE=InnoDB;

The record existing independently of the job is what makes every subsequent question answerable, and it is the difference between a delivery mechanism and a fire-and-forget dispatch. The payload is stored so that a replay does not need to reconstruct it from a source that may have changed.

Storing an excerpt of the subscriber’s response body is the field that removes most of the support load — a subscriber seeing their own 502 diagnoses it themselves. Truncating it is deliberate: an HTML error page is four kilobytes and there is no reason to keep all of it.

The retry schedule, published

documented on the same page as the payload schema:

  attempt 1  immediately      attempt 5  +1 hour
  attempt 2  +30 seconds      attempt 6  +6 hours
  attempt 3  +2 minutes       attempt 7  +24 hours
  attempt 4  +10 minutes      then       disabled, emailed

  total window: 31 hours. a timeout is 10 seconds.
  any 2xx is a success; everything else is a failure.

The total window matters more than the individual intervals, because a subscriber planning maintenance needs to know they have thirty-one hours rather than ten minutes. Stating what counts as success is the other half — a subscriber returning 302 needs to know whether that is acceptable, and it is not.

private const SCHEDULE = [0, 30, 120, 600, 3600, 21600, 86400];

$next = self::SCHEDULE[$delivery->attempts] ?? null;

if ($next === null) {
    $delivery->update(['state' => 'abandoned']);
    $this->endpoints->recordFailure($delivery->endpoint);

    return;
}

$delivery->update([
    'state' => 'pending',
    // jitter, so 4,000 deliveries do not retry in lockstep
    'next_attempt_at' => now()->addSeconds(
        $next + random_int(0, (int) ($next * 0.1))
    ),
]);

The jitter matters here for the same reason it matters everywhere: a subscriber who was down for an hour receives four thousand deliveries at the same instant when they recover, which knocks them over again. Ten per cent of the interval is enough to spread it and small enough not to affect the published schedule.

Signing, with a rotation path

$signed = ($timestamp = time()) . '.' . $payload;

$signatures = array_map(
    fn (string $s): string => 'v1=' . hash_hmac('sha256', $signed, $s),
    $endpoint->activeSecrets(),      // current, and previous
);

$headers = [
    'X-Signature'   => 't=' . $timestamp . ',' . implode(',', $signatures),
    'X-Delivery-Id' => $delivery->id,
];

Sending several signatures in one header is what makes rotation self-service: the subscriber accepts if any matches, so both secrets are valid during an overlap window and neither side has to coordinate a deploy. Publishing a thirty-day overlap turns a rotation from an event into a routine.

The timestamp in the signed string and in the header is what lets a subscriber reject a replayed request, and the documentation has to tell them to check it — a signature without a freshness window is valid forever to anybody who captured one.

Disabling a dead endpoint, loudly

if ($endpoint->consecutiveFailureDays() >= 7) {
    $endpoint->disable();

    Mail::to($endpoint->owner)->send(new EndpointDisabled(
        lastError: $endpoint->lastFailureSummary(),
        reenableUrl: URL::signedRoute('webhooks.reenable', $endpoint),
        backlogCount: $endpoint->pendingDeliveries()->count(),
    ));
}

Retrying forever against a dead endpoint is an unbounded queue, and disabling silently is worse than disabling loudly. The re-enable link is what keeps this out of the support queue, and it replays the backlog rather than starting from the next event — a subscriber who fixed their endpoint wants the week they missed.

Whether to retain the backlog at all is a storage decision that has to be made before the first endpoint fails. Fourteen days was chosen and it means a disabled endpoint can accumulate a substantial number of rows, which is bounded by the retention and is not bounded by anything else.

What a subscriber can see

GET /api/webhook-deliveries?endpoint=ep_88&since=2022-12-14
{
  "data": [{
    "id": "whd_9c1f4a7e",
    "event": { "id": "evt_8814", "type": "order.placed" },
    "state": "failed",
    "attempts": 5,
    "last_response": { "status": 502, "body": "<html>502 Bad..." },
    "next_attempt_at": "2022-12-14T10:41:02Z"
  }],
  "links": { "replay": "/api/webhook-deliveries/whd_9c1f4a7e/replay" }
}

A self-service delivery log and a replay endpoint removed the entire category of support ticket that prompted this, and the six hours of lost events from day four could have been recovered in one request. Building it after the incident rather than before is the ordinary outcome and the incident is what made the case.

Verifying it worked

# a subscriber that fails for six hours, in a harness
$ ./bin/webhook-drill --fail-for=6h --events=200
  delivered after recovery:      200
  abandoned:                       0
  peak concurrent deliveries:     14   (jitter working)

$ ./bin/webhook-drill --fail-for=8d --events=10
  endpoint disabled / emailed:  yes
  backlog retained:              10
  after re-enable:               10 delivered

Simulating a six-hour subscriber outage and asserting that everything is eventually delivered is the test that would have prevented the original ticket, and the peak concurrency figure is the assertion that the jitter works. The eight-day drill covers the disable-and-replay path, which is the one that is otherwise never exercised.

What this costs

A delivery table growing at the rate of the event stream, with a retention policy that has to be longer than the retry window and short enough to be affordable. Fourteen days against a thirty-one hour window is generous and produces a table that is one of the largest in the database within a quarter.

It is also a support surface. A delivery log and a replay endpoint are features with documentation, authentication and their own bugs, and they exist to reduce support load — which they do, and they are not free. The honest framing is that sending webhooks is a product rather than a job, and the first version being a job is what made that visible.