On a Tuesday evening, our message publishing pipeline locked up completely. Producers publishing transactional events began timing out after five seconds, worker queues starved, and web processes backed…
During an infrastructure audit of our legacy Beanstalkd instance, we found a tube named email-verification-reminders holding 1.8 million ready jobs. The worker service consuming it had been removed…