A webhook endpoint that survives the sender

The payment provider sent a settlement notification, our handler took nine seconds to process it, their timeout was five, and they retried. Within twenty minutes there were four hundred concurrent handlers all doing the same nine seconds of work for the same forty notifications.

The symptom

$ grep -c 'POST /webhooks/payments' access.log
1204

$ grep 'POST /webhooks/payments' access.log | awk '{print $NF}' | sort -n | tail -1
9.412

# distinct event ids in those 1,204 requests:
$ jq -r .event_id webhook-bodies/*.json | sort -u | wc -l
41

# 1,204 deliveries of 41 events, none acknowledged in time.

Their retry policy is exponential and generous, which means a handler that is slightly too slow produces a load pattern that guarantees it stays too slow. The system was doing thirty times the necessary work and completing none of it.

Why it happens

The handler was written as a controller action because that is what the route is, and controller actions do work and return a result. A webhook is not a request for work — it is a statement that something happened, and the sender needs an acknowledgement rather than an outcome.

The fix

Accept, verify, store, enqueue, return

public function __invoke(Request $request): Response
{
    // 1. verify — fast, and before anything is trusted
    $this->verifySignature($request);

    // 2. store the RAW body, so it is replayable
    $received = ReceivedWebhook::create([
        'provider' => 'payments',
        'payload'  => $request->getContent(),
        'headers'  => $request->headers->all(),
    ]);

    // 3. enqueue, and get out
    ProcessPaymentWebhook::dispatch($received->id);

    return response()->noContent();
}

Storing the raw body before parsing it is what makes the event recoverable when the handler has a bug — the work can be replayed from the stored bytes without asking the provider to resend, which for most providers is a support ticket.

Returning 204 for anything durably stored, including a payload that cannot be parsed, is the correct behaviour: a parse error is our problem and is not a reason for the provider to retry. Returning a 400 for a malformed body means their retry queue does our error handling.

Verifying against the raw bytes

private function verifySignature(Request $request): void
{
    $timestamp = (int) $request->header('X-Timestamp', '0');

    // a replay window, or a captured request is valid forever
    if (abs(time() - $timestamp) > 300) {
        abort(400, 'stale timestamp');
    }

    $expected = hash_hmac(
        'sha256',
        $timestamp . '.' . $request->getContent(),   // raw. never re-encoded.
        config('services.payments.webhook_secret'),
    );

    if (! hash_equals($expected, $request->header('X-Signature', ''))) {
        abort(400, 'bad signature');
    }
}

Re-encoding the parsed payload to compute the signature is the mistake that appears in half the implementations of this, and it works until a payload contains a non-ASCII character or a float — key order, unicode escaping and number formatting all differ between encoders.

The timestamp window is what makes a captured request expire. Without it, anybody who has ever seen a valid request can replay it indefinitely, and signature verification alone does not prevent that.

Some frameworks consume the input stream in middleware, which makes getContent return an empty string and produces a signature failure that looks like a secret mismatch. Checking that early saves an afternoon.

Idempotency, because retries are the normal case

public function handle(): void
{
    $event = json_decode($this->received->payload, true, flags: JSON_THROW_ON_ERROR);

    $claimed = ProcessedWebhookEvent::insertOrIgnore([
        'provider' => 'payments', 'event_id' => $event['id'],
    ]);

    if ($claimed === 0) {
        return;      // already handled. this is normal.
    }

    DB::transaction(fn () => $this->apply($event));
}

The claim and the work must be in one transaction, or a crash between them marks the event as processed without doing it — which is the failure that is silent and permanent. Using the provider’s event id rather than a hash of the body is what makes this work across their retries, since a retry may not be byte-identical.

Ordering, which is not guaranteed

what arrived, in this order, over four seconds:

  payment.refunded    (event 3, created 14:22:31)
  payment.captured    (event 2, created 14:22:29)
  payment.authorised  (event 1, created 14:22:28)

applying those in arrival order marks a refunded payment
as captured. the options:

  a state machine that rejects impossible transitions
  a sequence number, if the provider sends one
  refetching state from their API, ignoring the payload

The state machine is the answer that works with every provider and it makes out-of-order delivery a no-op rather than a corruption: a transition from refunded to captured is rejected and logged rather than applied. That is the difference between a bug report and a warning nobody has to act on.

Refetching from their API is the most robust option and turns the webhook into a signal to go and look, which removes ordering, replay and payload-tampering concerns in one move. It costs an API call per event and a rate limit to respect, and for anything financial it is worth both.

A dead letter path with an owner

public function failed(Throwable $e): void
{
    $this->received->update([
        'state' => 'failed', 'error' => $e->getMessage(),
    ]);

    WebhookFailed::dispatch($this->received);   // alert on ARRIVAL
}

// and the replay, which must be tested rather than assumed
// php artisan webhooks:replay --provider=payments --since=1h

Alerting on the first failure rather than on a queue depth is what catches a handler bug at event one instead of event four hundred. The replay command has to be exercised in a game day, because a message from three weeks ago may not deserialise against today’s code and finding that out during an incident is expensive.

Verifying it worked

# at their retry rate, with a deliberate handler failure
$ hey -n 500 -c 50 -m POST -D body.json 
    -H "X-Signature: $SIG" https://staging/webhooks/payments
  Requests/sec:  1841
  p95:           0.038s
  Status: [204] 500 responses

# and the duplicate case
$ for i in 1 2 3; do curl -s -o /dev/null -w '%{http_code} ' ...; done
204 204 204
$ mysql -Nse 'SELECT COUNT(*) FROM payments WHERE event_id = "evt_1"'
1

Load testing at the provider’s retry rate is the test that would have prevented the incident, and it is cheap to run once the handler returns immediately. The duplicate delivery check is the other half and asserts on the database rather than on the response.

The out-of-order case needs its own test, sending three events backwards and asserting the final state is correct. It is the one most likely to be skipped and the one whose failure is a wrong balance rather than an error.

What this costs

A queue between the notification and the effect, which means the effect now happens asynchronously and anything that assumed it was immediate has to be revisited. A user who completes a payment and lands on a page that reads the order state will occasionally arrive before the webhook is processed, and that is a real user-facing consequence requiring its own answer.

The stored payloads are also a growing table containing whatever the provider sends, which for a payment provider includes personal data and possibly card metadata. It needs a retention policy and a redaction pass, and both are easy to forget because the table is invisible until somebody asks about it during an audit.