This is a belated entry. It was intentionally delayed to try and maintain an advantage in a game of cat-and-mouse I was playing. Since the Super Happy Mega Fun Round is over now and I've returned to the usal day-in day-out dance, I can post this.
Sometime earlier this week, my personal server was compromised. For those who haven't been playing along at home, I have been doing hosting type things for a few years
( and can briefly recap how it all got started. )On Wednesday morning I was sitting down at work, sipping on my coffee and preparing for a long day of listening to people building shit with their mouths instead of with their hands. As usual routine I logged into the server and started skimming the mail that had collected since I'd gone to bed. The subject of one in particular grabbed my attention:
** URGENT *** it said.
Phishing attack on your server. I read on. The message warned that an attacker was using my server to harvest Amazon accounts and passwords. The informant included the phishing URL and so I was able to check things out and confirm -- sure enough, there was a very nasty set of web pages nestled into one of the subdirectories of a site I'd recently taken on hosting responsibilities for. I know the owner of the site and knew she hadn't done it -- especially since I hadn't yet given her access to the system to maintain her pages. I started getting a sense of low-grade concern, so I backed up the files for evidence and then wiped out the offending directory. Low-grade concern would later give way to frustration and outright dismay.
Okay, I've been compromised, I thought.
And I don't know how bad the attack was. Did they r00t it? Are other bad things going on? I better go find out. Thus began a 6 hour journey through the box. I found a couple of things right away that made my stomach sink: programs like top suddenly refused to run, citing dynamic links that had NEVER existed on the server. I kept digging and I eventually got my confirmation: an IRC "bouncer" program had been installed on the machine, listening on an unauthorized TCP port, disguised to look like my production MySQL daemon. Several key binaries like /bin/su and /bin/login had been replaced with trojans. One tool reported a number of cloaked processes running and loadable kernel modules installed, along with evidence of signatures for two prevalent "root kits."
The first law of running a system is that when you've been compromised, you do
not try to recover the system as it stands -- you can't be certain that you caught every little dastardly thing the attacker has done. The appropriate procedure is to back everything up for evidence if you can, back up your data files (so you can use them as incrementals to the weekly backups that you're doing... you
are doing weekly backups, right?) and then burn the whole damn system to the ground so you can do an install from scratch with clean, trusted media. The problem with this is that the machine in the hosting facility doesn't have a tape drive, and I'm not physically there so I can't swap media in and do an install myself. This momentarily stumped me until I started just making tarballs of everything important (all 24 websites that I host, all the mail files for my users, all the home directories of my users, etc etc) and pulling them off the system to a temporary storage facility. I didn't want to give my attacker any indication that I'd noticed him so, aside from removing the page to prevent further phish from being hooked, I left most everything in place until I was ready to have the system nuked. Wednesday night I was up late, making tarballs and FTPing them down to my PC (hooray for 6Mbps DSL lines and FileZilla!). I stayed home from work on Thursday because I still felt relatively crappy and worn down (small wonder, right?). Time was passed copying more files around between machines and trying to make sure I'd covered all my bases by backing up every last configuration file that I might need again. At around 2:30 PM or so, while talking with
shaddragon, I called it good and sent a service ticket to my hosting provider. In the letter I explained what had happened and gave their techs permission to burn the box down and do a complete reinstall. Shortly after the system suddenly went unreachable, which told me they'd yanked the machine from their network at the very least as a security precaution. All I could do was wait. Later that evening came a follow-up e-mail from the hosting provider -- my request to have
Fedora Core 4 installed (since they won't do FreeBSD) would cost me $150/hr since they don't provision Core 4 at this time, just Core 3. I talked with
points for a bit and he pointed me to an FC3->FC4 migration path that could be done remotely, which was perfect. I gave the representitive my blessing to do Core 3 and waited. And waited. And waited. Eventually I went to bed. Total cost of the day's effort? 19 some hours of my time, a bottle of vanilla Smirnoff vodka and three 2L bottles of Diet Sunkist. Oh yeah, and more hair from my head.
When I got up Friday morning I still wasn't feeling great, so I opted to stay home from work again. Good thing I did, because at around 10:30 in the morning I was contacted by the tech team to verify my request (I'd forgotten to give my authorization code) for the rebuild. Once they had the proper documentation they went to work, promising it would be done in "2 hours or so." I didn't get the system back until about 2:30 PM yesterday. I ran the migration and got the system up on an FC4 userland with an FC3 kernel. Since I wasn't about to go through all this suffering and NOT have the latest and greatest versions, I spent the following two hours and change wrestling with getting "yum" to work in a way that made sense to me. In the middle of the final bulk upgrade the system was suddenly halted by root, which severely pissed me off. Either I'd gotten hacked in the three hours since installation (mostly unlikely) or the hosting company had shut down my machine --
in the middle of a huge upgrade -- for no discernable reason. More waiting. Eventually the system came back and I went back to work. First I brought over the websites, then I had to screw around with the pre-installed Apache in order to make it run the way I wanted while supporting the features I needed. That was an adventure. As several of the websites I host depend on the DB back-end to make them go, I had to get the MySQL database system up and running next. This was a relatively painless install for me... but after all the work, I found Apache's PHP wouldn't talk with it.
twanfox was able to lend a bit of insight into that and I got things straightened out. With PHP and MySQL talking I could finally do an upgrade on the phpBB system that FrostFire uses (just to make sure I was current... again, with this much pain I deserve the newest and best). With the websites in place I turned my attention to getting the first of the hosted MUSHes back online, just to make sure I could get predictable behavior on this new OS. I had one small issue and then FrostFire was up and rolling again, so I started working on e-mail. That was at around 8pm last night. I worked on e-mail until 4:23 this morning, at which point I gave up and went to bed.
My desire to do
database-backed virtual mail accounts seems to have been a touch... optomistic. I'm in the process of giving it one last college try, then I'm going to fail back to the tried and true old way of just having a shell account for everyone who has e-mail on my system. Not my ideal way of handling it, but I can't have the mail system down much longer.
I've done things I know you'll never understand