The websocket layer used Redis pub/sub to fan out order updates to connected clients. It worked perfectly on a single instance and, after the cluster migration, every publish was being delivered to all six nodes regardless of who was subscribed — because that is what classic pub/sub does in a cluster, by design.
The symptom
$ redis-cli --stat
------- data ------ --------------------- load --------------------
keys mem clients blocked requests connections
412008 1.88G 418 0 88104102 (+41208) 9412
# 41,208 requests per second across the cluster, at a
# publish rate of about 6,800/s.
$ redis-cli -h node-04 INFO stats | grep instantaneous
instantaneous_ops_per_sec:6841
# node-04 has no subscribers. it is receiving every
# message anyway.Six thousand eight hundred publishes producing forty-one thousand operations is the cluster broadcasting each one to every node. The nodes with no subscribers do the work of receiving and discarding, which is most of the traffic on the cluster bus.
Why it happens
A channel name is not a key, so it does not hash to a slot and no node owns it. To guarantee that a subscriber connected to any node receives a message published to any other, the cluster propagates every publish to every node.
That is the only correct behaviour given the semantics, and it means pub/sub throughput does not improve with cluster size — it degrades, because every added node is another recipient.
The fix
Sharded channels
# classic: every node, every publish
SUBSCRIBE orders.updated
PUBLISH orders.updated '{"id":8814}'
# sharded: the channel name hashes to a slot, and only the
# nodes serving that slot are involved
SSUBSCRIBE 'orders.updated.{shard-3}'
SPUBLISH 'orders.updated.{shard-3}' '{"id":8814}'
# the hash tag {shard-3} is what pins it to a slot,
# exactly as it does for a key.The hash tag is the mechanism and it means a channel name is now a sharding decision. A single sharded channel for everything puts all the traffic on one node and looks like a regression, which is the naive migration and is the first thing everybody tries.
Choosing a channel name that distributes
// wrong: one channel, one slot, one node
$redis->spublish('orders.updated', $payload);
// also wrong: a channel per order, and a subscriber that
// must subscribe to thousands
$redis->spublish("orders.{{$order->id}}", $payload);
// what worked: shard by the thing the SUBSCRIBER groups by
$shard = crc32((string) $order->customer_id) % 64;
$redis->spublish("orders.{shard-{$shard}}", $payload);
// 64 channels, distributed across the slot space, and a
// subscriber computes its own shard from the customer it
// is watching.Sharding by the customer rather than by the order is what makes the subscriber side tractable: a websocket connection watching one customer subscribes to one channel rather than to one per order. The number of shards is a fixed constant, which means it is a decision that cannot be changed without a coordinated restart of every subscriber.
Sixty-four was chosen so that a cluster of six nodes has roughly ten channels each and can grow to sixteen nodes before the distribution becomes lumpy. That arithmetic is worth doing rather than picking a round number, because a shard count below the node count leaves nodes idle.
The subscriber side, which is where the work is
// a subscriber must connect to the node that OWNS the
// slot — the client cannot follow a MOVED redirect for a
// subscription
$node = $cluster->nodeForSlot(
$cluster->slotForKey("orders.{shard-{$shard}}")
);
$connection = $cluster->connectionTo($node);
$connection->ssubscribe("orders.{shard-{$shard}}", $handler);
// and on a resharding or a failover, the slot moves and
// the subscription has to be re-established. there is no
// automatic migration.The lack of automatic re-subscription on a slot move is the operational cost and it is not documented prominently. A failover silently stops delivering to that subscriber, which presents as a subset of users not receiving updates — a failure that is invisible in every metric except the one measuring delivery.
Watching for cluster topology changes and re-subscribing is application code that has to be written, and getting it wrong in the other direction — re-subscribing too eagerly — produces duplicate delivery. That is why the client library support lagged the server by months.
Whether this was the right tool at all
the question the migration prompted:
pub/sub is fire-and-forget. a subscriber that is
disconnected for 200ms misses the message entirely,
with no way to know.
for a websocket layer where a missed update means a
stale price on somebody's screen, that had been
acceptable and nobody had decided it was.
a stream with a consumer group gives:
delivery tracking, replay, and a pending list
at the cost of:
memory, trimming, and a consumer that must ack
we kept pub/sub. it was the right call and it was the
first time anybody had made it.Reaching a decision that had previously been an accident is most of the value of a migration like this. Fire-and-forget is correct for a UI update that will be corrected by the next one, and is wrong for anything where a missed message is a lost fact — and the two had been served by the same mechanism for three years.
Verifying it worked
$ redis-cli --stat
keys mem clients requests
412008 1.88G 418 9204188 (+7102)
# 41,208/s → 7,102/s, at the same publish rate.
$ for n in 01 02 03 04 05 06; do
> printf 'node-%s %sn' "$n"
> "$(redis-cli -h node-$n INFO stats | grep -oP 'ops_per_sec:K.*')"
> done
node-01 1204
node-02 1188
node-03 1211
node-04 1180
node-05 1162
node-06 1157
# evenly distributed, which is the assertion.The even distribution across nodes is what proves the hash tags are working — a shard count that collides would show as two busy nodes and four idle ones. The overall reduction from forty-one thousand to seven thousand operations per second is the six-fold saving that the broadcast had been costing.
What this costs
A channel scheme that is now a design decision with a fixed constant in it, and changing the shard count means every publisher and every subscriber agreeing simultaneously. That is a coordinated restart of the whole websocket layer, which is why the number was chosen with room to grow rather than to fit today.
The subscriber-side topology handling is the larger cost and is application code that would not exist with a single instance. A failover that silently stops delivery to a subset of users is a failure mode created by this change, and the only defence is a delivery metric per shard — which is another thing to build and to watch.