A profile taken on the machine that was actually slow

Three days of profiling locally to explain a production slowdown that did not exist locally.

local, the same request:  41ms
production:              880ms

what differed, in order of size:

  the database had 400× the rows          620ms
  the cache was cold for this key         180ms
  opcache was full and restarting          31ms
  the CPU was slower                        8ms

none of which a local profile can show, and all of
which a sampled production profile showed in
twenty minutes.

A local profile measures the code and a production profile measures the system, and a slowdown that only exists in production is by definition not about the code. Three days is the honest cost of the instinct to reproduce first; the sampled profiler had been available the whole time and felt like a bigger commitment than it was.