A profile of the queue worker rather than the request

Months of profiling the web path, and a queue worker consuming three times the CPU that nobody had ever looked at.

  web tier    4 hosts × 20 workers, profiled monthly
  queue tier  6 workers, never profiled

  CPU seconds a day:
    web       41,208
    queue    128,104

and the top entry in the queue profile: a JSON encode
of a 200 KB payload, twice per job, because the payload
was encoded to log it and then encoded again to send it.

Profiling follows attention and attention follows user-visible latency, which means the asynchronous half of a system is systematically under-examined. The double encode had been there for two years and was found in the first twenty minutes of looking — the finding is less interesting than the fact that nobody had looked.