queues

  • A consumer that had to be idempotent and finally was

    A handler documented as idempotent since 2022, which was not, and a redelivery that proved it. A docblock claiming idempotency is a claim and not a mechanism, and…

  • A dead letter queue that reached zero

    A dead letter queue that had held between forty and eight thousand messages since 2019, empty for the first time. An empty dead letter queue is not the…

  • An outbox relay with a second instance, partitioned

    A single relay process that had become the throughput ceiling, split into two by partitioning on the aggregate identifier. The modulo on a hash of the aggregate id…

  • Message payloads reduced to identifiers, one year on

    A change made in 2024 to send identifiers rather than snapshots, and what a year of it produced. The lookup cost is trivial when the consumer shares a…

  • Message payloads that had grown to 200 KB

    A message carrying the full order including every line, every address and the customer record, because it was easier than a lookup. Fat messages are defended as avoiding…

  • An outbox relay that stalled and told nobody

    A relay process that died on an unhandled exception and left rows accumulating with no error anywhere. The second alert is the one that mattered: the relay stopped…

  • A job with a unique lock that outlived the job

    A job marked unique so it cannot be queued twice, and a worker killed mid-run leaving the lock held for its full duration. The hour was chosen when…

  • A licence change, and the fork we did not take

    20 March, a licence change, and a fork announced within a week. A dependency that is no longer open source, in production, everywhere.

  • A consumer group rebalance during a deploy

    Every deploy restarting six consumers at once, triggering a rebalance that stops consumption for forty seconds. Rolling the restarts is the obvious answer and it is slower, which…

  • The service we merged back

    Two services extracted in 2021, a team of four maintaining four pipelines, and forty-one of forty-four releases touching both.