An outbox, and the publisher we removed from the transaction

Twice in one week an order was committed and the event announcing it was never published, so the fulfilment side never heard about it. The publish was inside the database transaction, which looks like the careful thing to do and is the cause.

The symptom

DB::transaction(function () use ($order) {
    $order->place();
    $order->save();

    $this->broker->publish(new OrderPlaced($order->id()));   // ← here

    $this->stock->reserve($order->lines());   // throws sometimes
});

// two failure modes, both real:
//   publish succeeds, transaction rolls back
//     → an event for an order that does not exist
//   publish fails, transaction would have committed
//     → the whole order is lost to protect a message

the two incidents:

  Tue  the broker connection timed out. the exception
       rolled back a valid order. the customer was
       charged and had no order.
  Thu  stock reservation failed after a successful
       publish. fulfilment picked an order the
       database had rolled back.

both directions of the same bug, three days apart.

Why it happens

Two systems and one commit cannot be made atomic without a distributed transaction, and nobody wants a distributed transaction. Putting the publish inside the database transaction feels like it buys atomicity and buys the opposite: it couples the durability of the order to the availability of the broker.

The fix

The outbox table

CREATE TABLE outbox (
  id           BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
  aggregate_id BINARY(16) NOT NULL,
  type         VARCHAR(64) NOT NULL,
  payload      JSON NOT NULL,
  published_at DATETIME(6) NULL,
  attempts     TINYINT UNSIGNED NOT NULL DEFAULT 0,
  created_at   DATETIME(6) NOT NULL,
  KEY idx_unpublished (published_at, id)
) ENGINE=InnoDB;

DB::transaction(function () use ($order) {
    $order->place();
    $order->save();

    $this->outbox->record(
        aggregateId: $order->id(),
        type: 'order.placed',
        payload: $order->toEventPayload(),
    );                                    // a row, not a network call

    $this->stock->reserve($order->lines());
});

The transaction now contains only database writes, which is the only thing it can guarantee. A rollback removes the outbox row along with the order, and a commit makes both durable together — the atomicity the original code was reaching for, achieved by not involving the second system.

The relay

$rows = DB::table('outbox')->whereNull('published_at')
    ->orderBy('id')->limit(200)->get();

foreach ($rows as $row) {
    $this->broker->publish($row->type, $row->payload, [
        'message-id' => (string) $row->id,
    ]);

    DB::table('outbox')->where('id', $row->id)
        ->update(['published_at' => now()]);
}

Publishing and then marking is the order that matters: a crash between the two republishes on the next pass, which is a duplicate. The reverse order would lose the message, and a duplicate is recoverable while a loss is not — which is why the consumer has to be idempotent and why the outbox row id is sent as the message identifier.

The index that makes the poll cheap

the poll runs every 200ms, on a table with 4 million
rows of which 3 are unpublished.

MySQL orders NULLs first in an ascending index, so the
unpublished rows are at the head of idx_unpublished. the
query reads three index entries and stops: 0.4ms.

without the index: 4,102,884 rows, 1.8 seconds, five
times a second.

Ordering, and where it actually matters

// global ordering is not needed and is expensive.
// per-aggregate ordering is, and comes free from the id.
$this->broker->publish(..., [
    'partition-key' => $row->aggregate_id,
]);

// messages for one order land on one partition, in order.
// messages for different orders may interleave, which
// nothing depends on.

Requiring global ordering would force a single relay with no concurrency; requiring per-aggregate ordering allows several relays partitioned by aggregate. We run one because the volume does not need more, and the partition key means adding a second is a configuration change rather than a redesign.

The lag metric

// the number that matters is age, not count
$oldest = DB::table('outbox')
    ->whereNull('published_at')
    ->min('created_at');

$gauge->set('outbox_oldest_unpublished_seconds',
    $oldest === null ? 0 : now()->diffInSeconds($oldest));

// alert above 60 seconds for 5 minutes.
// a count of 4,000 that clears in 2 seconds is fine.
// a count of 1 that is 20 minutes old is an incident.

A depth metric would have said nothing here for the same reason it says nothing about a queue: four thousand messages draining at two thousand a second is healthy, and one message stuck for twenty minutes is not. The age is directly comparable to the promise, which is that an event is published within a minute.

Cleanup

-- published rows, older than 7 days, batched
DELETE FROM outbox
WHERE published_at IS NOT NULL
  AND published_at < DATE_SUB(NOW(), INTERVAL 7 DAY)
LIMIT 5000;

-- 7 days because that is how long a replay might
-- reasonably be asked for. the domain_events table is
-- the permanent record; this one is a buffer.

Verifying it worked

# a deliberate rollback after the outbox write
$ ./bin/outbox-drill --fail-after-record
  orders created:        0
  outbox rows:           0
  messages published:    0        # correct

# a broker outage during the relay
$ ./bin/outbox-drill --broker-down=120s --orders=500
  orders created:      500        # unaffected
  published after
    recovery:          500
  duplicates:            3        # a crash mid-batch
  consumer effect:       0        # idempotent

# steady state, one month
  oldest unpublished, p99:  1.8s
  alerts:                   1 (a broker restart, 4m)

Five hundred orders created during a two-minute broker outage is the outcome the whole change is for — the order path no longer has the broker in it at all. The three duplicates are expected and are the reason the consumer’s idempotency is not optional.

What this costs

A delay between commit and publish, which is under two seconds at the ninety-ninth percentile and is not zero. Anything that expects a message to have arrived by the time the HTTP response is written is now wrong, and one place did — a test that placed an order and immediately asserted on a consumer’s side effect.

It is also a component that can stall silently. A relay that dies leaves rows accumulating with no error anywhere except the lag metric, which means the metric is load-bearing rather than informational — and a monitoring gap here looks exactly like the original bug it replaced.