Well it had to happen at some point, today I had a nice email from DigitalOcean saying they’ve disabled networking on one of my servers. This was because it’s ip address had been reported to RBL’s by several other servers.
Looking at the logs they included I was beating the s**t out of others wp-admin login pages. Now I know I wasn’t doing it personally, it was the first time in a long time I was in bed early and this seemed to start at 2am.
Luckily I could access my Droplet using the Console page, so after login I sat thinking ‘um…..’ where exactly do you start. The server normally has quite a bit of traffic so the logs are always cluttered. Needle in a haystack springs to mind.
I decided to run htop and see if the server was doing much without any traffic coming in. Oh yes /usr/bin/host is eating resources. So do I kill it or not. I decided not to at this point. Without networking I’m not doing anymore harm, and leaving it running may help find out what’s calling it.
It was a good call. I can’t give details of everything I did, I spent a hour hours checking through stuff. I do remember checking lsof and finding a link between a process id for host and a file within wpallimport’s uploads directory. So I had a look in there, followed by some further searching of google. 1 file in particular .sd0 seems to bring back results and this seems to be what’s caused it.
To get my server running again, I disabled the entire site within apache that was affected (luckily not a major site) and reboot the server. Once I was happy there were no cronjobs or anything calling on this script I mailed DigitalOcean and asked them to re-enable networking. They’re pretty speedy and within 15 mins had done it. A further reboot and my servers back online minus the one site I’ve disabled.
I expect the cleanup for this is going to take weeks of checking files, against backups while keeping as much as possible online.
I’m pretty confident I know what’s caused it, an out of date wordpress install with an out of date wpallimport install. It really goes to show that you have to check old stuff and keep it upto date.
The most annoying thing for me is that WordPress has a multisite option (which I use on 2 installs) and this allows me to keep plugins and everything upto date easily of sub-sites that are barely used. but it doesn’t extend to multiple domains which would really allow wordpress to be used across all my domains from one central console and then everything would be kept upto date in one go.
I know there’s a plugin for multisite domains, but I feel this is more of a hack of the wordpress system rather than wordpress properly designed to function with this in mind. I don’t want to install it and encounter many more problems.
It’s very bad admin on my part not having kept this site upto date, I’ll be the first to admit that but it’s easy to forget about installs you don’t use regularly. There must be some kind of nagios plugin to alert me to out of date plugins/versions for wordpress so I’ll be looking for that later in the week 🙂