Hardening a public-facing server before it gets found

The stack from building this box assumed a machine nothing had found yet. That assumption lasted about four days. This is what the log looked like at the end of the first week, and what it took to make it stop mattering.

The symptom

$ grep 'Failed password' /var/log/auth.log | wc -l
4127

$ grep 'Failed password' /var/log/auth.log 
>   | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -5
   1841 203.0.113.44
    902 198.51.100.17
    611 203.0.113.9
    288 192.0.2.55
    140 198.51.100.203

Four thousand attempts in six days, against root, admin, oracle, postgres and a long tail of names nobody has used since 2003. None of them succeeded, because the password was long. That is the entire defence, and it is one leaked password away from being no defence.

Why it happens

The IPv4 address space is small enough to scan continuously, and it is scanned continuously. There is no period during which a new server is unknown; the only variable is how long it takes for the first scanner to reach that particular address, and the answer is hours. Obscurity is not part of the threat model — moving SSH to port 2222 reduces log noise and nothing else.

The fix

Keys, and actually turning passwords off

Adding a key makes key login work. It does not stop password login from also working, which is the step that gets skipped — and until it is done, the key has changed nothing about the server’s exposure.

# /etc/ssh/sshd_config
PasswordAuthentication no
ChallengeResponseAuthentication no
PermitRootLogin no
AllowUsers deploy
UsePAM yes

ChallengeResponseAuthentication is the line people miss. Left on, keyboard-interactive can still accept a password through PAM even with PasswordAuthentication no, which produces a server that looks locked down and is not.

A firewall that defaults to closed

Ubuntu ships with no firewall rules, so every service that binds to a public interface is reachable. That includes MySQL if its bind-address was ever changed, and Redis, which in 2015 still has no authentication worth the name and will happily accept commands from anyone who can reach it.

$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow OpenSSH
$ sudo ufw allow 'Nginx Full'
$ sudo ufw enable

$ sudo ufw status numbered
[ 1] OpenSSH      ALLOW IN    Anywhere
[ 2] Nginx Full   ALLOW IN    Anywhere

Default deny is the important half. An allow-list of three ports on a default-allow policy protects nothing, because the next service installed will bind to a fourth.

With the rules in place it is worth asking what was reachable before them. A default Ubuntu install with this stack on it usually has more listening than expected, and each entry is either deliberate or a mistake:

$ sudo ss -lptn
State   Local Address:Port   Process
LISTEN  0.0.0.0:22            sshd
LISTEN  0.0.0.0:80            nginx
LISTEN  0.0.0.0:3306          mysqld     <-- public
LISTEN  127.0.0.1:6379        redis-server

MySQL bound to every interface is the common one, and the firewall now blocks it — but defence in depth means fixing the binding as well, because the firewall is one ufw disable away from being absent. Setting bind-address = 127.0.0.1 in my.cnf means the port is not open even if the rules are.

fail2ban, and not only for SSH

fail2ban watches a log for a pattern and adds a firewall rule when it repeats. It is nearly always installed for SSH and left there, while the same brute force runs against every HTTP login on the box.

# /etc/fail2ban/jail.local — never edit jail.conf
[sshd]
enabled  = true
maxretry = 4
findtime = 600
bantime  = 86400

[nginx-http-auth]
enabled  = true
logpath  = /var/log/nginx/error.log
maxretry = 5
findtime = 600
bantime  = 3600

findtime is the parameter that decides whether this is useful or hostile: four failures in ten minutes is an attack, four over a week is someone who changed their laptop. Put the configuration in jail.local, because jail.conf is replaced on package upgrade.

monit for the things that fall over

Hardening is not only about intruders. A PHP-FPM pool that dies at three in the morning is an outage regardless of cause, and on a one-person project nobody is watching.

check process nginx with pidfile /var/run/nginx.pid
    start program = "/usr/sbin/service nginx start"
    stop program  = "/usr/sbin/service nginx stop"
    if failed host 127.0.0.1 port 80 protocol http
        then restart
    if 3 restarts within 5 cycles then alert

check filesystem rootfs with path /
    if space usage > 85% then alert

The restart-count alert is what keeps this honest. A process restarted once fell over; a process restarted five times in five cycles is crash-looping, and only the second is worth an email. The disk check is there because a full disk is the most common way a small server dies, and it does so silently — MySQL stops accepting writes and nginx keeps serving cached pages.

Patching, without a maintenance window

The last piece is the one that is easiest to postpone indefinitely. An Ubuntu box left alone for six months is a liability regardless of how tightly the firewall is drawn, because the vulnerability will be in a package rather than in the configuration.

// /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};

Unattended-Upgrade::Mail "[email protected]";
Unattended-Upgrade::Automatic-Reboot "false";

Security origins only — pulling in every update unattended is how a minor release changes a default at three in the morning. Automatic-Reboot stays off because a kernel update rebooting unannounced is worse than a kernel update waiting until someone is awake; the mail is what tells you one is pending.

Verifying it worked

# one week later
$ grep 'Failed password' /var/log/auth.log | wc -l
0

$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Currently failed: 2
|- Total failed:     318
`- Currently banned: 11

The attempts have not stopped — they never stop. They now fail at the firewall after four tries instead of continuing indefinitely, and password authentication is not available to succeed at even if one of them guessed correctly.

What this costs

Locking yourself out is now a real failure mode, and it will happen. The recovery plan is worth writing down before it is needed: the provider’s serial console, a second key belonging to someone else, and a fail2ban ignoreip for the office address so a mistyped passphrase at a bad moment does not ban the only person who can fix it.

The other cost is that none of this addresses the application. A server with no password login and a tight firewall is still fully compromised by one SQL injection in a form, and that is the more likely route. This is the floor, not the ceiling.