The box falls over at about sixty concurrent requests. Not gradually — the load average goes to forty, everything starts swapping, and the only thing that brings it back is restarting Apache. The reflex answer is a bigger box, and it is the wrong answer here, because the memory is being spent on something that does not need it. I have made this move once before, on a single-site VPS with a four-line rewrite config. This one is a five-year-old application with nine .htaccess files and a rewrite block nobody fully understands, and that is what makes it a different job rather than a repeat of the same one.
The symptom
$ ps -C apache2 --no-headers | wc -l
398
$ ps -ylC apache2 --sort:rss | tail -3
S 33 21847 1 0 80 0 28912 71204 - ? 00:00:02 apache2
S 33 21903 1 0 80 0 29104 71204 - ? 00:00:01 apache2
S 33 21955 1 0 80 0 29488 71204 - ? 00:00:03 apache2
$ free -m
total used free buffers cached
Mem: 7986 7802 184 21 402
-/+ buffers/cache: 7379 607
Swap: 4095 1876 2219
$ uptime
14:22:31 up 61 days, load average: 38.44, 33.10, 21.87Three hundred and ninety-eight Apache processes at roughly 28 MB resident each. That is more memory than the machine has, which is why 1.8 GB of it is in swap, and swapping is what turns a slow site into a stopped one. The load average has not been under five for three days.
The second number worth having is what those processes are actually doing, and the status module answers it in one request:
$ curl -s 'http://localhost/server-status?auto' | head -6
Total Accesses: 1841266
Total kBytes: 40218114
BusyWorkers: 398
IdleWorkers: 2
ReqPerSec: 34.9
BytesPerReq: 22368Twenty-two kilobytes per request against a page that returns four kilobytes of HTML. The overwhelming majority of what this server does is hand over images, stylesheets and scripts.
Why it happens
Apache is running the prefork MPM, because mod_php is not thread-safe and prefork is the only supported way to run it. Prefork means one process per concurrent connection. mod_php means every one of those processes carries a PHP interpreter with the extensions loaded, whether or not the request in front of it has anything to do with PHP.
So a request for a 4 KB logo occupies a 28 MB process for as long as the client takes to receive it, and on a mobile connection that is not brief, because prefork holds the worker until the response has been fully written. The arithmetic is the entire problem: sixty concurrent PHP requests is fine, four hundred concurrent requests of which three hundred and forty are images is fatal, and the second is what one page with forty assets on it produces.
# /etc/apache2/apache2.conf
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 400
MaxRequestsPerChild 0
</IfModule>MaxClients 400 on 8 GB was never a survivable number — four hundred times 28 MB is 11 GB. Lowering it would have stopped the swapping and converted the outage into a queue, which is better and still not a fix. The fix is that serving a logo should not cost 28 MB.
The fix
PHP-FPM first, still behind Apache
The first stage does not involve nginx at all. PHP moves out of Apache into its own pool, reached over a socket, with Apache still in front. That keeps the change reversible in one config line, and it banks most of the memory win before anything risky happens.
; /etc/php5/fpm/pool.d/shop.conf
[shop]
user = www-shop
group = www-shop
listen = /var/run/php5-fpm-shop.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 24
pm.start_servers = 6
pm.min_spare_servers = 4
pm.max_spare_servers = 10
pm.max_requests = 500
pm.status_path = /fpm-status
slowlog = /var/log/php5-fpm-shop.slow.log
request_slowlog_timeout = 5s# /etc/apache2/conf.d/php-fpm.conf — mod_fastcgi, Apache 2.2
FastCgiExternalServer /var/www/php5-fcgi -socket /var/run/php5-fpm-shop.sock -idle-timeout 60
AddHandler php5-fcgi .php
Action php5-fcgi /php5-fcgi
Alias /php5-fcgi /var/www/php5-fcgia2dismod php5 is the line that actually changes anything. With mod_php still loaded Apache keeps an interpreter in every process and the stage has achieved nothing. Resident size per Apache process fell from 28 MB to a little under 3 MB the moment it was unloaded, and the swap usage went with it.
nginx in front, and the .htaccess
Stage two takes the longest and almost none of it is nginx configuration. It is the rewrite rules.
Apache reads .htaccess on every request, from every directory along the path. That is a per-request filesystem cost, and it is also why these rules were allowed to accumulate: anyone could add one, in any directory, with no reload and no review. nginx has no equivalent. Every rule has to be found, understood and rewritten into a location block in a file only root can edit.
$ find /var/www/shop -name .htaccess | wc -l
9
$ grep -c Rewrite /var/www/shop/public/.htaccess
31RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?_route=$1 [QSA,L]
RewriteRule ^catalogue/([0-9]+)/?$ product.php?id=$1 [L]
RewriteRule ^old-catalogue/(.*)$ /catalogue/$1 [R=301,L]
<FilesMatch ".(ini|log|sql)$">
Deny from all
</FilesMatch>That is the readable third of the file. Translated, with the rest of the server block around it:
server {
listen 80;
server_name shop.example www.shop.example;
root /var/www/shop/public;
index index.php;
location / {
try_files $uri $uri/ /index.php?_route=$uri&$args;
}
location ~ ^/catalogue/([0-9]+)/?$ {
try_files $uri /product.php?id=$1;
}
location ^~ /old-catalogue/ {
rewrite ^/old-catalogue/(.*)$ /catalogue/$1 permanent;
}
location ~* .(ini|log|sql)$ {
deny all;
}
location ~ .php$ {
try_files $uri =404;
fastcgi_pass unix:/var/run/php5-fpm-shop.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~* .(jpg|png|gif|css|js|ico)$ {
expires 30d;
access_log off;
}
}Caveat
try_files $uri =404; inside the PHP location is not decoration. Without it nginx hands any path ending in .php to FPM, so a file uploaded as an image and stored under the document root as avatar.php is executed. This is the most common way a working nginx configuration is also a remote code execution hole, and no test that only requests real pages will find it.
Three rules did not survive translation cleanly, and they are the ones that cost the afternoon. RewriteCond %{REQUEST_FILENAME} !-f becomes try_files, which checks in order and falls through — close, but not identical, and an existing directory with no index behaves differently. Apache’s [QSA] has to be reproduced by appending the query string by hand. And a .htaccess in a subdirectory has no equivalent at all: the four under uploads/ happened to be nothing but Deny from all and collapsed into one block, which was luck rather than skill.
Pool sizing and OPcache
Every tutorial picks pm.max_children out of the air. The number that matters is the resident size of a worker under this application’s load, and it takes one command to measure.
$ ps --no-headers -o rss -C php5-fpm | awk '{s+=$1; n++} END {print s/n/1024, n}'
41.7 24Forty-two megabytes each, on a box that can spare about 5 GB for PHP once MySQL and the operating system have taken theirs. That is 119 workers on paper, and it is nonsense: 119 concurrent PHP requests would exhaust the database connection limit long before the memory ran out. A pool is sized by whichever ceiling is lower, and here that is MySQL. It went to 40, with pm.max_requests = 500 left in place because a five-year-old codebase leaks and recycling a worker is cheaper than finding out where.
OPcache went on in the same change. PHP 5.5 has shipped it since June and it stays off until you enable it, which is easy to miss when the extension already appears in php -m.
; /etc/php5/fpm/conf.d/05-opcache.ini
zend_extension=opcache.so
opcache.enable=1
opcache.memory_consumption=192
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=8000
opcache.revalidate_freq=60
opcache.fast_shutdown=1revalidate_freq=60 rather than 0 is a deployment decision dressed as a performance setting: a deploy is now invisible for up to a minute unless reloading FPM is part of the deploy. It became part of the deploy.
Cutover
No DNS involved, and no waiting for propagation. nginx takes port 80, Apache moves to 8080 and keeps running, and the rollback is swapping the two ports back and restarting both — about fifteen seconds, at any hour.
$ nginx -t
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ service apache2 restart
$ service nginx start
$ curl -sI http://127.0.0.1:8080/ | head -1 # apache, still there
HTTP/1.1 200 OK
$ curl -sI http://127.0.0.1/ | head -1 # nginx, now public
HTTP/1.1 200 OKKeeping Apache alive on 8080 for a fortnight was worth more than it cost. Twice in that fortnight a page turned out to behave differently under nginx, and being able to fetch the Apache version of the same URL in one command is the difference between a bug report and a diagnosis.
Verifying it worked
Three measurements, in ascending order of honesty. Throughput at the concurrency that used to kill the box; then memory; then the error log, which is the only one of the three that finds the rewrite rules that did not survive.
# before — apache + mod_php
$ ab -n 2000 -c 60 http://127.0.0.1/catalogue/
Requests per second: 38.11 [#/sec] (mean)
Time per request: 1574.4 [ms] (mean)
Failed requests: 41
# after — nginx + php-fpm
$ ab -n 2000 -c 60 http://127.0.0.1/catalogue/
Requests per second: 214.63 [#/sec] (mean)
Time per request: 279.5 [ms] (mean)
Failed requests: 0
$ free -m | sed -n '2,4p'
Mem: 7986 4102 3884 64 2611
-/+ buffers/cache: 1427 6559
Swap: 4095 0 4095Swap is empty and 2.6 GB is in the page cache, which is where memory on a web server is supposed to be. The throughput figure is the least interesting of the three: most of the gain is nginx serving static files without waking PHP at all, so it compares two different amounts of work.
$ awk '$9 == 404 {print $7}' /var/log/nginx/access.log
| sort | uniq -c | sort -rn | head -3
412 /feeds/google-merchant.xml
88 /catalogue/print/1841
31 /sitemap-products.xmlThree rewrite rules had been missed, all of them in .htaccess files below the document root, and all three served something a machine consumed rather than something a person clicked — a feed, a print view, a generated sitemap. Nobody would have reported any of them. The first day of the 404 log found all three in about a minute, which is the argument for reading it on the first day rather than the first week.
What this costs
.htaccess is gone, and with it the ability for anybody other than root to add a redirect. That has been sold to me before as pure gain and it is not: a marketing redirect that used to take somebody ninety seconds now takes a request, a config change, an nginx -t and a reload. It is genuinely safer and genuinely slower, and if nobody accounts for the second half, the rules stop being added properly and start being handled by whatever plugin can do it from an admin screen.
There is also a second daemon. nginx has its own configuration language, its own log format, its own upgrade cycle and its own way of failing, and twelve years of Apache knowledge does not transfer to any of it. Both now have to be monitored, patched and backed up — and /etc/nginx has to be added to the backup set, which is exactly the sort of thing that is remembered right up until the restore.
The failure mode changed as well, which is easy to file as a win and should not be. With a fixed pool, request forty-one queues in the socket backlog instead of being served slowly, so the site no longer degrades evenly — it is fast for most people and a clean 502 for the rest. That is a better failure and a different one, so the alerting has to move with it: the listen queue on the FPM status page is now the number that predicts an outage, and load average is not.