Installing fail2ban is treated as a step that protects the server. What it actually gives you is one jail — [ssh], watching auth.log — while the same brute force is running against wp-login.php in the nginx access log and SMTP AUTH in the mail log, neither of which anything is reading.
# /etc/fail2ban/jail.local — never jail.conf, which is replaced on upgrade
[ssh]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 4
[wp-login]
enabled = true
port = http,https
filter = wp-login
logpath = /var/log/nginx/access.log
maxretry = 5
findtime = 300
bantime = 3600
# /etc/fail2ban/filter.d/wp-login.conf
[Definition]
failregex = ^<HOST> .* "POST /wp-login.php
ignoreregex =A jail is a filter, a log path and an action bound together, so it watches exactly one file — which means the unit of configuration is the log, not the machine. The setting that quietly wastes the effort is port: the ban is written into iptables for the ports the jail names, so an HTTP attacker banned by a jail that says port = ssh carries on hitting the login form unimpeded. Test a new filter with fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/wp-login.conf before enabling it, because a regex that matches nothing fails silently and the jail simply never bans anyone. And behind a proxy or a CDN the address in the log is the proxy’s, so the jail will eventually ban that instead.