The RabbitMQ cluster that deadlocked on unroutable confirms

On a Tuesday evening, our message publishing pipeline locked up completely. Producers publishing transactional events began timing out after five seconds, worker queues starved, and web processes backed up until Nginx returned 504 Gateway Timeouts. RabbitMQ CPU usage was idling at eight percent, memory alarms were green, and disk space had over two hundred gigabytes free. The cluster was perfectly healthy and totally unresponsive.

The symptom

20:14:00  Deploy ships a new billing event publisher.
20:14:12  Worker connection count increases from 40 to 850.
20:14:30  Publish latency jumps from 1.4ms to 5,000ms (timeout threshold).
20:15:00  RabbitMQ management API shows:
          connections: 1,240 (state: blocked / flow)
          messages_unacknowledged: 0
          queue_depth: 0
20:15:30  Publishers throwing AmqpTimeoutException: "Waiting for confirm timed out".

Publishers were configured with confirm_select(true) and the mandatory flag. When a new event type was published to an exchange where the binding key was misspelled, the broker could not route the message to any queue. Instead of discarding it, the broker triggered an unhandled return path that deadlocked the synchronous confirm loop.

Why it happens

When publishing with both publisher confirms and the mandatory flag in PHP AMQP extensions, an unroutable message sends a basic.return frame immediately followed by a basic.ack. If the application’s AMQP client does not register an explicit basic_return callback, the client driver blocks waiting for the confirmation seqno, buffering inbound frames until the socket buffer fills. Once the socket buffer fills, TCP flow control stalls the entire connection.

The fix

Registering return listeners and alternative exchanges

final class ReliableAmqpPublisher
{
    private AMQPChannel $channel;

    public function __construct(AMQPChannel $channel)
    {
        $this->channel = $channel;
        $this->channel->confirm_select();

        // Handle unroutable messages asynchronously without blocking confirms:
        $this->channel->set_return_listener(
            function (int $replyCode, string $replyText, string $exchange, string $routingKey, AMQPMessage $message) {
                error_log(sprintf(
                    'Unroutable AMQP message [%s] on exchange [%s] with key [%s]',
                    $replyText,
                    $exchange,
                    $routingKey
                ));
            }
        );
    }

    public function publish(string $exchange, string $routingKey, string $payload): void
    {
        $msg = new AMQPMessage($payload, ['delivery_mode' => 2]);
        $this->channel->basic_publish($msg, $exchange, $routingKey, true);
        
        // Wait with a finite timeout and drain return frames
        $this->channel->wait_for_pending_acks_returns(2.0);
    }
}

In addition to the driver-level return listener, we declared an Alternate Exchange (AE) policy on all production exchanges. If a message has no matching binding key, RabbitMQ automatically routes it to an unroutable.events dead-letter queue rather than dispatching a return frame back to the client.

Verifying it worked

# Run chaos test publishing 10,000 unroutable messages on staging:
$ ./bin/console amqp:test-unroutable --count=10000

Results:
  Published: 10,000
  Published duration: 1.82s (5,494 msgs/sec)
  Timed out confirms: 0
  Messages captured in unroutable.events queue: 10,000
  Publisher connections blocked: 0

The alternate exchange absorbed all ten thousand unroutable events without a single publisher blocking or throwing a timeout exception.

What this costs

Storage management for unroutable message queues. Messages caught by an alternate exchange must have TTL policies or an alert on depth, otherwise forgotten events accumulate silently and consume disk space.

And slight latency overhead. Calling wait_for_pending_acks_returns per message has a higher CPU cost than fire-and-forget publishing. For high-volume producers, batching confirms in groups of fifty messages is necessary to maintain throughput.