An idempotent consumer for a broker that redelivers

A payment capture message was delivered twice during a broker failover and the handler charged the card twice. The broker was behaving correctly — at-least-once delivery is what it promises — and the handler had been written by somebody who had read that promise and had not internalised it.

The symptom

$ redis-cli XPENDING payments workers - + 10
1) 1) "1668421262-0"
   2) "worker-3"
   3) (integer) 412008      # idle time, ms
   4) (integer) 2           # DELIVERY COUNT

$ mysql -Nse "SELECT gateway_reference, COUNT(*)
  FROM captures GROUP BY gateway_reference HAVING COUNT(*) > 1"
ch_9c1f4a7e   2
ch_8c9e1140   2

# two captures, two gateway charges, one order each.

A delivery count of two on a pending message is the broker saying it has delivered this once before and had no acknowledgement. That is not an error condition and the handler had no branch for it.

Why it happens

Exactly-once delivery is not available from any broker worth using, because it requires a distributed transaction between the broker and the consumer’s side effects. Every broker offers at-least-once and documents it, and the documentation is read at the point of choosing the broker rather than at the point of writing a handler.

The fix

The claim, in the same transaction as the work

DB::transaction(function () use ($message) {
    $claimed = DB::table('processed_messages')->insertOrIgnore([
        'handler'    => self::class,
        'message_id' => $message->id,
        'claimed_at' => now(),
    ]);

    if ($claimed === 0) {
        return;      // already handled. this is normal.
    }

    $this->capture($message);
});

The claim and the work must be in one transaction or a crash between them leaves the message marked processed and not done, which is the failure that is silent and permanent. insertOrIgnore against a unique index is the arbitration — a SELECT followed by an INSERT has the same race the whole exercise exists to remove.

Scoping the claim to the handler class matters when several handlers consume the same message, and using the broker’s message id rather than a hash of the payload is what survives a redelivery that is not byte-identical.

The half that is not in the database

// the transaction covers the local write, not the gateway
// call — which is a second system
$charge = $this->gateway->capture(
    $message->token,
    $message->amount,
    // the gateway's OWN key, derived from something stable
    idempotencyKey: "capture-{$message->orderId}",
);

Capture::create([
    'order_id'          => $message->orderId,
    'gateway_reference' => $charge->reference(),
]);

Two layers of the same mechanism is more machinery than it sounds and it is the only arrangement that is actually correct: the claim table stops the handler running twice, and the gateway key stops the charge happening twice if the handler crashes after the gateway call and before the local write.

Deriving the gateway key from the order id rather than generating one is the detail that makes it work across a redelivery — a fresh key on each attempt is the original bug with an extra API parameter.

Natural idempotency, and when no table is needed

naturally idempotent — repeating changes nothing:
  setting a field, deleting by id, upserting a search
  document by id, invalidating a cache key

not idempotent:
  incrementing a counter, sending an email, charging a
  card, and DISPATCHING ANOTHER JOB ← the subtle one

the last is why an otherwise-idempotent handler still
needs a claim: a redelivery produces two follow-up jobs,
and the duplicate propagates down the chain.

The dispatch case is the one that gets missed because the handler itself looks safe — it sets a field and enqueues the next step, and setting a field twice is harmless. The second follow-up job is the duplicate, one hop further away from the cause.

Poison messages, and the delivery count

// a message that crashes the worker never reaches the
// handler's retry logic, so the limit has to come from
// the BROKER's delivery count
if ($message->deliveryCount() >= 5) {
    $this->deadLetter($message, 'delivery count exceeded');
    $this->ack($message);

    return;
}

// Redis streams:  XPENDING reports it
// SQS:            ApproximateReceiveCount
// RabbitMQ:       an x-death header, after a DLX round trip

A retry limit held in the consumer resets when the consumer restarts, which is exactly what happens when a message kills it — so the limit has to be a property of the message. Every broker exposes this differently and none of them make it prominent, which is why the usual way to discover it is finding a queue that has been retrying the same message for a week.

Ordering, which is a separate problem

// idempotency does not give ordering. these can arrive
// in any order:
//   payment.authorised  (created 14:22:28)
//   payment.captured    (14:22:29)
//   payment.refunded    (14:22:31)

// so the handler is a state machine, and an impossible
// transition is rejected rather than applied
match ([$payment->state, $event->type]) {
    ['authorised', 'captured'] => $payment->capture(),
    ['captured',   'refunded'] => $payment->refund(),
    default => $this->logOutOfOrder($payment, $event),
};

A state machine makes out-of-order delivery a logged no-op rather than a corruption, which is the difference between a warning nobody has to act on and a refunded payment marked as captured. It is the answer that works with every broker, where sequence numbers depend on the broker providing them.

Verifying it worked

public function testDuplicateDeliveryChargesOnce(): void
{
    $m = new CapturePayment(id: 'msg-1', orderId: 8814, ...);

    $this->handler->handle($m);
    $this->handler->handle($m);      // the redelivery
    $this->handler->handle($m);

    $this->assertCount(1, Capture::where('order_id', 8814)->get());
    $this->gateway->assertCaptureCount(1);
}

public function testOutOfOrderDeliveryIsRejected(): void
{
    $this->handler->handle($this->refunded());
    $this->handler->handle($this->captured());

    $this->assertSame('refunded', $this->payment()->fresh()->state);
}

# and the chaos test, which is the one that matters
$ ./bin/broker-chaos --kill-worker-mid-handler --messages=2000
  delivered:     2,412 (412 redeliveries)
  captures:      2,000
  gateway calls: 2,000
  divergences:   0

Killing the worker mid-handler is what exercises the crash-between-claim-and-work path, and it is the failure the transaction exists to prevent. Four hundred and twelve redeliveries producing two thousand captures is the assertion.

What this costs

A table that grows at the rate of the message stream and needs a retention policy, which has to be longer than the broker’s maximum redelivery window or a very late redelivery is processed again. Fourteen days was chosen against a broker that retries for seven, and the reasoning belongs in a comment on the migration.

The two-layer idempotency is also a coupling to the gateway’s implementation of it. A provider whose idempotency keys expire after twenty-four hours gives no protection against a redelivery on day three, and finding that out means reading their documentation rather than assuming — which is a check that belongs in the integration’s own notes and is easy to omit.