Personal Server "Offline"

[Modred]

I put offline in quotes, because everything is actually running; however, Apache is denying all inbound requests. The SSH server is still running and I still have access to the machine, but I'm having trouble getting Apache back in line. I'm afraid that my machine may have been compromised in some manner.

The access log shows around 50 hits last night from a Chinese IP address, issuing HTTP CONNECT requests to pop3.sohu.com over a course of around 4 minutes.

When I attempt to stop the server, everything appears fine. Upon restarting the server, I get an error that it is already running with a PID of 622. A quick glimpse at the running processes shows 622 as a mysqld process, not httpd, so there's another suspicious behavior.

I'm really not sure what's going on here, and hope someone here has some idea on how to first fix this and then fortify against it happening again. As a note, I built this Apache install, along with mod_php and mod_python that it's using, from source, so it may not have all of the best security features enabled. If anyone has tips for rebuilding these products to be well secured, please share as well.

UPDATE:
I issued a killall on mysqld and it took care of those suspicious processes. Then restarted Apach and MySQL, and it looks fine, but I still have no idea what happened. Assuming this was a security breach, I definitely need to harden some security.

[TimP]

A common entry point is SSH brute forcing as well as FTP brute forcing. Check /var/log/auth.log (or similar) to see all the authentication attempts.

Here's a small excerpt from my public facing server:

Apr 29 14:00:02 mybox sshd[17115]: Invalid user staff from 200.91.87.212
Apr 29 14:00:03 mybox sshd[17117]: Invalid user sales from 200.91.87.212
Apr 29 14:00:04 mybox sshd[17119]: Invalid user recruit from 200.91.87.212
Apr 29 14:00:05 mybox sshd[17121]: Invalid user alias from 200.91.87.212
Apr 29 14:00:06 mybox sshd[17123]: Invalid user office from 200.91.87.212
Apr 29 14:00:08 mybox sshd[17127]: Invalid user tomcat from 200.91.87.212
Apr 29 14:00:09 mybox sshd[17129]: Invalid user webadmin from 200.91.87.212
Apr 29 14:00:10 mybox sshd[17131]: Invalid user spam from 200.91.87.212
Apr 29 14:00:11 mybox sshd[17133]: Invalid user virus from 200.91.87.212
Apr 29 14:00:12 mybox sshd[17135]: Invalid user cyrus from 200.91.87.212
Apr 29 14:00:13 mybox sshd[17137]: Invalid user oracle from 200.91.87.212
Apr 29 14:00:14 mybox sshd[17139]: Invalid user michael from 200.91.87.212
Apr 29 14:00:16 mybox sshd[17143]: Invalid user test from 200.91.87.212
Apr 29 14:00:16 mybox sshd[17145]: Invalid user webmaster from 200.91.87.212

So within 14 seconds, I got 14 brute force SSH logins. There are several ways to stop this. The solution I use is to require public key authentication, in other words, no keyboard interactive (typed in password) login and I have a copy of my private key on machines I need to connect from.

If you want to set this up, in sshd_config make sure you have these lines:

PubkeyAuthentication yes
# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no

sshd disconnects you off before you can type a password if you don't have a public key. The downside is you can't connect to it from arbitrary machines without your private key.

Another option is the denyhosts (http://denyhosts.sourceforge.net/) script. It's a pretty decent compromise between usability and security. It will add brute forcing IPs to a host denylist so they can no longer connect to the server after, for example, 15 failed logins. If you notice in my log excerpt, all those authentication attempts were from the same IP address. Most distros have a denyhosts package tailored to the distro's setup.

A "security through obscurity" measure would be to change the port that sshd runs on, either through sshd_config or through port redirection on the router level. A lot of brute force scripts are dumb and are hard coded to attack port 22.

The last thing I can think of is to setup iptables to only allow connections from one IP address (yours, presumably :)). I have a "deskside" Linux server that I only connect to from one machine (my desktop) and by virtue of my university's network policy, I have to run it internet facing (I used to just run it behind a router firewall). The only downside is if your IP address changes you get locked out of your box. In my case the IP address is fairly static and I can always login via serial port if it decides to hide on me.

Also check last (the command last) and see the login times. A good intruder will cover their tracks and delete any evidence of their login, but it's worth taking a look at. Also check /etc/passwd and make sure no new users were added by the intruder. I'd also change all your login passwords for good measure.

EDIT: And for completeness, are you listening on any other ports besides 22 and 80? Possibly 3306 for MySQL?

[Modred]

Checking last and /etc/passwd showed no unusual behavior, and while I didn't find a /var/log/auth.log, I did find some other files that appeared to be a similar function and it only showed myself logging in.

My server is behind a router, and at the time of the incident, ports 21 (FTP), 22 (SSH) and 80 (HTTP) were being forwarded to the machine, so if MySQL is listening for connections (which it probably is, since I have the server running and haven't messed with my.ini), it should only be visible on the local network. I've since closed down the FTP forwarding, since I wasn't using it anyway.

There have been a few bug fix releases since I put all of this stuff together, so I'll probably recompile Apache, Python, and mod_python along with any other stuff I've got going (just redid mod_php recently, so I'll probably let it be). I doubt any of those bugs were to blame for any problem, but might as well patch up those holes before they become a problem.

I may set up the key based authentication, since I generally only use one or two computers to access the shell, so it wouldn't be a big deal to put a key on each of them. As far as the iptables solution: that would certainly keep people out, but I don't think I need a sledge hammer like that, yet.