On a Tuesday evening, our message publishing pipeline locked up completely. Producers publishing transactional events began timing out after five seconds, worker queues starved, and web processes backed…
For two years our Monit daemon was configured to restart any queue worker process that exceeded 500MB of RAM. The restart hid a memory leak that slowly grew…
A long-running queue worker daemon processing invoice generation began crashing every evening around midnight with ErrorException: fopen(): Too many open files. The process restart limit in systemd eventually…