For the first time (at least the first time that I'm aware of), one of my Linux servers got hacked recently.  Thankfully, this wasn't a production server, it was a contingency box.

I detected the break-in because the Nagios console displayed a large number of processes and the load average had increased beyond the threshold (meaning the box was much busier than usual).  Once I got into a position to log onto the box, there were a huge number of processes running a program called 'bruteforcessh'.  Also there was evidence that the wunderbar_emporium Linux kernel exploit had been attempted.  Whilst I didn't think the kernel version installed on this box was vulnerable to this attack, a new user had been created on the box, which couldn't have been done without some kind of privilege escalation.

After a bit of troubleshooting, I came to the conclusion that the intruder had managed to brute-force the password for the Nagios user account, which obviously wasn't strong enough.  The security log at /var/log/secure indicated that the user had been able to log in using a password.  The scary thing was that I think the initial break-in attempt happened several weeks ago judging by the dates / times in the log.  From there, they had then run a privilege escalation attack, created a new user and installed some software that appeared to be designed to propagate the virus by attacking other neighbouring servers.  It was also running an IRC client, which was presumably a means of communicating with other servers.

The moral of the story is don't ever allow password authentication on a server that is publicly accessible by SSH.  Ensure that public key authentication is enabled and that password authentication is disabled and particularly that root login is not permitted.  These are the options you need in /etc/ssh/sshd_config to set this up:

PasswordAuthentication no
PubkeyAuthentication yes         # this is the default
PermitRootLogin no

This makes it statistically almost impossible to log into the server by brute force methods.  To ensure that you can log into a server with a key exchange, firstly, generate a key-pair:

$ ssh-keygen

Then, you can add the public key to your authorized_keys file on the remote servers to which you wish to log in.

$ ssh [server-name] "echo `cat $HOME/.ssh/id_rsa.pub` >> "' $HOME/.ssh/authorized_keys'

This command adds the locally generated public key to the list of authorised keys that can log into a remote server.  If the .ssh directory or authorized_keys file does not exist, on the target server, you'll need to create and permission them as follows:

$ ssh [server-name]
Password:
$ mkdir .ssh
$ chmod 700 .ssh
$ touch .ssh/authorized_keys
$ chmod 600 .ssh/authorized_keys
$ exit

You should then be able to log in using your public key rather than a password.  Look at ssh-agent (Linux / UNIX) or Pageant (Windows) to save you retyping your key passphrase for each login.

The particular server that was attacked is going to have to be rebuild from scratch as it is always hard with privilege escalation attacks to assess to what extent the server was compromised and I'd rather reinstall everything to ensure that all compromised code is removed.