Two services, one HTTP call between them, and an incident report that said the checkout service was down when it had never stopped running. It was waiting. The service it called had a slow afternoon, and because the call was synchronous, its slow afternoon became everyone’s.
The symptom
Checkout returned 502 for eleven minutes. Its own health check was green throughout, its database was idle, and the logs showed requests arriving and never completing.
[14:22:07] POST /orders → calling inventory.reserve
[14:22:37] POST /orders ← cURL error 28: Operation timed out after 30001ms
[14:22:38] POST /orders → calling inventory.reserve
[14:23:08] POST /orders ← cURL error 28: Operation timed out after 30001ms
$ ps aux | grep php-fpm | wc -l
52 # pm.max_children = 50. all of them waiting.Fifty workers, each holding a connection open for thirty seconds against a service that was answering in four minutes rather than not answering at all. A timeout bounds how long one request waits; it does nothing about how many are waiting at once.
Why it happens
A synchronous call makes the caller’s availability the product of both services rather than its own. Two services at 99.9% chained give 99.8%, and that is the optimistic version — it assumes failures are independent, and a shared database or a shared network makes them correlated.
The question worth asking at each call site is narrower than it looks: does the caller need the answer in order to respond to its own client? For inventory reservation the honest answer was no. The order was accepted either way; the reservation was bookkeeping that had to happen, not bookkeeping that had to happen first.
The fix
Publish the fact, do not request the action
The shape of the change is smaller than the architectural language around it. The caller stops asking a service to do something and starts stating that something happened.
// before — checkout tells inventory what to do, and waits
$response = $this->http->post('http://inventory/reserve', [
'json' => ['order_id' => $order->id, 'lines' => $order->lines()],
'timeout' => 30,
]);
if ($response->getStatusCode() !== 200) {
throw new ReservationFailed();
}
// after — checkout states what happened, and stops caring
$this->events->publish(new OrderPlaced(
$order->id,
$order->lines(),
$order->placedAt()
));The naming carries the decision. ReserveInventory is a command with an implied recipient and an implied answer; OrderPlaced is a statement of fact with neither. Once the message is a fact, adding a second consumer — the warehouse, the analytics pipeline — requires no change to the publisher at all.
The events that must not be events
This is not a transformation to apply everywhere, and the failure mode of overapplying it is worse than the problem it solves. Anything the caller needs an answer to stays synchronous.
// stays synchronous — the customer is waiting for this answer
$auth = $this->payments->authorise($card, $order->total());
// stays synchronous — the response depends on it
$price = $this->pricing->quote($basket, $customer->tier());
// becomes an event — nobody is waiting, and a retry is harmless
$this->events->publish(new OrderPlaced(/* ... */));
// the test: if the consumer runs an hour late, is the system wrong,
// or just behind? "wrong" means it has to stay synchronous.That last comment is the whole heuristic. Behind is a latency problem and can be monitored. Wrong is a correctness problem and cannot be fixed by a queue.
Idempotency, which is not optional
Every broker worth using delivers at least once, which means a consumer will see the same message twice — on a redelivery after a crash, or when an acknowledgement is lost. A consumer that decrements stock is a consumer that will eventually decrement it twice.
final class ReserveStock
{
public function handle(OrderPlaced $event)
{
// the unique index does the work; the check is the fast path
try {
$this->db->table('reservations')->insert([
'order_id' => $event->orderId,
'created_at' => $event->placedAt,
]);
} catch (QueryException $e) {
if ($this->isDuplicate($e)) {
return; // already handled. ack and move on.
}
throw $e;
}
$this->stock->decrement($event->lines);
}
}The constraint is doing the deduplication rather than a SELECT before the INSERT, because two workers can pass the same check concurrently. Distinguishing a duplicate-key error from every other query error is the fiddly part and is worth a named method rather than a string match inline.
Ordering, which is usually not the problem people expect
The first objection raised in review is always ordering, and it is almost always the wrong thing to worry about. A single queue with one consumer is ordered; the moment there are two consumers for throughput, it is not, and no amount of configuration fixes that without giving up the throughput.
// the fragile design: two events whose order matters
new OrderPlaced($id);
new OrderLinesAdded($id, $lines); // useless if it arrives first
// the robust one: the event carries what the consumer needs
new OrderPlaced($id, $lines, $placedAt);
// and where order genuinely matters, a version the consumer checks
if ($event->version <= $this->lastSeenVersion($event->orderId)) {
return; // stale. discard.
}Designing events to be self-contained removes most ordering requirements outright. Where one genuinely remains — a status that moves forward and must never move back — a version number the consumer compares is simpler and more robust than trying to make the transport ordered.
The messages that fail forever
A consumer that throws on a malformed message will be redelivered that message immediately, throw again, and occupy the queue in a loop that produces nothing but log volume. A dead letter queue after a bounded number of attempts is the smallest thing that prevents it.
$ rabbitmqctl list_queues name messages consumers
orders.placed 0 4
orders.placed.retry 2 4
orders.placed.dead 7 0 ← these need a human
# the dead queue having a consumer count of zero is deliberate.
# it is an inbox, not a stage.The dead letter queue only works if something alerts on its depth. Left unmonitored it is a place where lost orders accumulate silently, which is materially worse than the synchronous call that at least failed loudly.
Verifying it worked
# stop the consumer entirely and place an order
$ docker stop inventory-consumer
$ curl -s -o /dev/null -w '%{http_code} %{time_total}n'
-X POST https://shop.internal/orders -d @order.json
201 0.184
$ rabbitmqctl list_queues name messages
orders.placed 1
$ docker start inventory-consumer
$ sleep 2 && rabbitmqctl list_queues name messages
orders.placed 0
# and the duplicate check
$ rabbitmqctl ... republish the same message
$ mysql -e 'SELECT COUNT(*) FROM reservations WHERE order_id = 91204'
1Killing the consumer and watching the producer keep working is the assertion the whole change exists to make true. The duplicate republish is the second one, and it is the check most likely to be skipped because at-least-once delivery is easy to believe in and hard to remember.
What this costs
Eventual consistency is now visible to the business, and that conversation belongs to them rather than to engineering. Somebody will ask what happens if an order is placed and stock is not reserved for four seconds, and the answer — that it can oversell by a small amount under specific conditions — is a policy decision, not a bug. Having it explicitly is far better than the previous arrangement, where the same overselling happened during any outage and nobody had described it.
Operationally there is now a broker to run, monitor and back up, and a class of problem that did not exist before: a consumer that is up, reading, and processing more slowly than the producer publishes. Queue depth becomes a first-class metric, and a rising one is the earliest warning available — earlier than latency, earlier than errors. That is worth something, but it is one more thing that has to be watched by someone.