Month: August 2016

I hate being woken at 2am, to be told “we’re under attack!”. Well that’s pretty much this morning 🙁

Now to be fair, it wouldn’t have even been noticed. Our servers and setup handled it very well and it was only noticed during a switch over to a backup server with less resources during maintenance.

On checking the logs I could see lots of attempts like

X.Y.X.Y - - [28/Aug/2016:03:12:16 +0100] "POST /wp-login.php HTTP/1.0" 200 5649 "-" "-"

We’re walking 114,893 requests to just the one server. My first instinct is to add the offending IP address to our IPTables BLOCK list and just drop the traffic altogether. Except this no longer works with CloudFlare since the IP addresses the firewall will see is theirs not the offender!

No problem, CloudFlare deals with this(!?) I can just turn on ‘Under Attack’ mode and let them handle it. This is where you quickly learn it doesn’t really do much. Personally I got a lovely 5 second delay when accessing our websites with ‘Under Attack’ activated. but our servers were still being bombarded with requests. So I added the offending IP addresses to the firewall on CloudFlare. Surely that will block them from even reaching our servers! While I can’t say it had no effect, out of a number of IP addresses I had added some were still hitting our servers quite heavily.

So the question becomes ‘How can I drop the traffic at nginx level?’. Nginx is configured to restore the real IP addresses, so I should be able to block the real offenders here not CloudFlare.

Turns out to be pretty easy. Add:

### Block spammers and other unwanted visitors ###
include blockips.conf;

Into the http section in /etc/nginx/nginx.conf

http {

### Block spammers and other unwanted visitors ###
include blockips.conf;

...

}

Then create the /etc/nginx/blockips.conf:

deny X.Y.X.Y;

Just add a deny line for each offending IP. Then reload nginx (service nginx reload). I’d recommend testing your new config first (nginx -t)

Now with that done of both servers I’m still seeing requests but they are now getting 403 errors and not allowed to hit any of the websites on our servers 🙂 after another 5 minutes of attack they clearly gave up as all the requests stopped.

But we’re not done. What if they’re just changing IP addresses? we need this to be automatic. Enter Fail2Ban. I haven’t used this in some time but I know what I need it to do.

1. Check all the websites log files for wp-login.php attempts
2. Write to /etc/nginx/blockips.conf
3. Reload nginx

Should be quite simple. Turns out it is, but it took hours trying to figure out how since all the guides seem to think you’ll be using Fail2Ban to configure IPTables.

Here’s the files/configuration I added for Fail2Ban

# WordPress brute force auth filter
#
# Block IPs trying to auth wp wordpress
#
# Matches e.g.
# X.Y.X.Y - - [28/Aug/2016:03:12:16 +0100] "POST /wp-login.php HTTP/1.0" 200 5649 "-" "-"
#

[Definition]
failregex = ^<HOST> .* "POST /wp-login.php.*200
ignoreregex =</pre>

/etc/fail2ban/jail.d/wordpress-auth.conf
<!--?prettify linenums=true?-->
<pre class="prettyprint">[wordpress-auth]
enabled = true
filter = wordpress-auth
action = wordpress-auth
logpath = /var/log/nginx/*access*.log
bantime = 1200
maxretry = 8

# Fail2Ban configuration file based on dummy.conf
#
# Author: JD
#
#

[Definition]

# Option: actionstart
# Notes.: command executed once at the start of Fail2Ban.
# Values: CMD
#
actionstart = touch /etc/nginx/blockips.conf
service nginx reload

# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
#
# dont do this actionstop = echo "" > /etc/nginx/blockips.conf
# service nginx reload
actionstop =

# Option: actioncheck
# Notes.: command executed once before each actionban command
# Values: CMD
#
actioncheck =

# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionban = echo "deny <ip>;" >> /etc/nginx/blockips.conf
service nginx reload

# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionunban = sed -i "/<ip>/d" /etc/nginx/blockips.conf
service nginx reload

[Init]

init = 123

It’s pretty simple and will need some tweaking. I’m not so sure 8 requests in 20 mins is very good. We do have customers on some of our sites who regularly forget their password. The regex does look at the 200 code, I read that a successful auth would actually give a 304. Not sure if this is correct so will need some testing. I also found other information on how to change the code to a 403 for failed login attempt. I think this would be a huge plus, but I’m not looking into that tonight.

A few tests using curl and I can see Fail2Ban has done it’s job, added the IP to the nginx blockips file and reload nginx 😀 I’m not too worried about syncing this across servers, as shown tonight they will balance well and I’m confident that within a few minutes of being bombarded they would both independently block the IP.

So there’s my morning of working around using CloudFlare but still keeping some form of block list. Hope this helps someone. Please comment to let me know if you found any of this useful/worth reading.