That Time I Got Hacked

It happened sometime in the early 2000’s. Probably 2001 or 2002.

It was very early on in my career as a UNIX system administrator. At the time I was responsible for the shared web server environment at Indiana University. I was an apache admin on Tru64 UNIX and Solaris with hundreds of departmental accounts. Thankfully I don’t have to do that anymore!

From my first days of working with computers I was always interested in security. And I ran OpenBSD on my desktop, which billed itself as “The World’s Most Secure Operating System”.

One day I installed a fresh copy of the OS, finishing around 4:00 PM, and went home not too long after that.

I came in the next morning and got back to work setting things up and configuring the system. I remember that I went to add a firewall rule and wasn’t able to save the file containing the pf ruleset. After just trying again a couple more times, I realized that the file was immutable, which was definitely weird. But I thought there was maybe something I didn’t yet understand about OpenBSD and pf. Then I noticed that the file had a new rule at the very top allowing all packets to pass from any source to any destination. Now I knew for sure that I had not added that rule. And when I saw it, I froze. It was super creepy, and I knew I’d been pwned.

In OpenBSD you can set the immutable bit on a file in multiuser mode at runlevel 3, but to remove it you have to go down to single user more at runlevel 1. So I did just that, fixed it, and brought the machine back up to removed the stupid firewall rule.

I couldn’t find any more suspicious files anywhere in the filesystem, but I was really nervous about doing anything else on the box because obviously it had just been compromised at the root level. So I went and asked for help from a friend who had a lot more experience than me. He helped me investigate. We found irc running and killed the process. But it kept respawning every time we rebooted. We searched and searched and couldn’t find anything out of place.

After a bit of trial and error, my buddy helped me add a couple lines of bash to /etc/rc.conf that added logging to the init process. We used logger to simply record the complete path of each process spawned from init and its child processes. We rebooted. It did the trick and revealed a hidden directory named “.\ “. That’s dot backslash space. A file or directory name that starts with a dot is hidden from the default view of the directory, and this one’s name was just one space. The backslash was there to escape the space so that the shell wouldn’t interpret it as a delimiter. The hidden directory was tucked away somewhere in /var/run or /var/lib. So if you didn’t know what you were looking for, it’s very unlikely that you would ever notice it.

Finding the attacker’s hidden directory was like winning the lottery for me. They had left behind their entire rootkit. It included the original attack tool which exploited a vulnerability in sshd known at the time as CVE 2001–0144. There was also a syslog cleaning tool, the eggdrop irc server, a network interface sniffer, and a bunch of trojaned binaries including ifconfig which is what was actually spawning eggdrop. So when it booted and came up into multiuser mode with networking, ifconfig would bring up the interfaces and spawn the rogue service.

At that point we also had modification timestamps on the hidden directory and all it’s files. And we could see the box had been rooted at about 6:00 PM, only two hours after I had installed it. It’s still amazing to me how quickly it all happened.

Back then, OpenBSD wasn’t nearly as easy to update as Mac or Windows is now. It required a manual process of patching source code with the patch utility, and then recompiling large parts of the operating system and the ports tree. Still though, there was no excuse for me to go home that day without patching the OS first.

So that’s what happened. Here are my lessons learned.

1. Always apply the latest updates immediately after you install the OS. If I had done that, this attack wouldn’t have been successful.
2. Always scope your firewall rules. There is just no need to have SSH wide open to the whole world unless you’re building a bastion host. If I had done that, the attacker wouldn’t have even been able to see that I had sshd running at all.

Even the world’s most secure operating system can get pwned if you make n00b mistakes like I did.