Months of profiling the web path, and a queue worker consuming three times the CPU that nobody had ever looked at.
web tier 4 hosts × 20 workers, profiled monthly
queue tier 6 workers, never profiled
CPU seconds a day:
web 41,208
queue 128,104
and the top entry in the queue profile: a JSON encode
of a 200 KB payload, twice per job, because the payload
was encoded to log it and then encoded again to send it.
Profiling follows attention and attention follows user-visible latency, which means the asynchronous half of a system is systematically under-examined. The double encode had been there for two years and was found in the first twenty minutes of looking — the finding is less interesting than the fact that nobody had looked.