Everyone wanted the site to be fast and nobody could say what fast meant, so the alert threshold was a round number somebody picked. The order matters because…
Forty gigabytes of logs a day and no answer to “is checkout slow”. Metrics, logs and traces answer different questions, and logs were doing all three badly.
Fifty PHP-FPM workers all waiting on one slow upstream is a site that is down because of a service most of its pages do not use. Rejecting immediately…
The database password was in the repository, then in the image, then in an environment variable readable by anything that could run ps. Where it went next.