The box I use to host this blog, plus my own instance of Confluence and Jira, is a humble-but-dedicated Linux server running CentOS 5, hosted by Serverbeach. Yesterday, I got an abuse report that a number of other boxes had been getting automated password scans — originating from my server’s IP address. Uh-oh! Had someone compromised my box?

I opened up an SSH client, and logged in to my server as each of the named users. The bash welcome message showed just what I’d suspect — last login at some reasonable time, from an IP known to me. UNTIL I logged in as the “nagios” user, and discovered that the last login was on December 22, from “ac9ed6e3.ipt.aol.com”. UH-OH. I’ve been PwN3d.

It looks like someone guessed the password for the “nagios” user I created when I was setting up a server monitor. It probably didn’t occur to me that I was creating a public-facing login when I created the “nagios” user, and used something easy to guess. Crap! What’d they do while they were in there?

I pulled the .bash_history file for that user, which you can see here in its entirety, if you’re interested. Unless the user edited the .bash_history file as a red herring, it looks like they downloaded a password scanner utility to /tmp/.k (a dot-prepended directory, so it’d be hidden unless you used ls -a), then fired it up to scan the first two octets of my IP range. And then came back periodically to check results using “screen”.

Here’s what “ps -u nagios” showed:

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
nagios   30769  0.0  0.0  2588  736 ?        S    Oct17   0:10 ntpd
nagios   23380  0.0  0.1  3668 1276 ?        S    Nov03   0:59 ntpd
nagios   16926  0.0  0.1  6100 1036 ?        Ss   Dec22   0:00 SCREEN
nagios   16927  0.0  0.1  5404 1396 pts/3    Ss+  Dec22   0:00 /bin/bash

The first two processes, 30769 and 23380, are, I think, Nagios doing its regular thing. But the other two processes were spawned by the uninvited user — a “SCREEN” session, and a login shell.

I quickly changed the password for the “nagios” user, then killed all the “nagios” user’s processes and deleted everything in /tmp/.k. I ran “sudo rpm -Va” to see if any of my packages had been, you know, sneakily altered, but my expertise runs out there.

As a professional, especially as one who depends on others to execute Big Chair Sysadmin tasks, I wouldn’t put a client’s SSH front door out there in the open, where anyone can come knocking. I always request a firewall in front, which usually only allows SSH logins from a particular (or, even better, private network) IP. Get a VPN connection to the hosting provider’s network, and it’s reasonably secure and portable. So this is pretty much a case of the cobbler’s children going shoeless – oy!

Serverbeach doesn’t offer a firewall solution, so I’m going to lock down the SSH on the box myself. Anyone care to offer an opinion as to whether you prefer IP restriction (not all that portable; I’m often on various wireless connections), certificate restriction (spiffier, but more confusing), or some other stealthy methods like changing the default port?