Admin Heroics

You know, 99% of the time, we have a pretty boring job. Sometimes we get to work on interesting problems, or maybe a system goes down, but for the most part, it's pretty mundane.

Sometimes, though, we get called to do relative heroics. Before I was even an admin, I did tech support for an ISP in West Virginia. Once, the mail server went down hard. 20,000 people around the state suddenly had no email, and the two administrators weren't able to be contacted. I was the only guy in the office who knew linux, and it just so happened that I had the root password to that server because I helped the younger admin a few weeks earlier with something.

I reluctantly agree to take a look at the thing, having never touched QMail (ugh), I delved into the problem. Numerous searches later led me to conclude that a patch would (probably?) fix the problem. I explained that to my bosses, and that I had never done anything like this before, but that I thought I might be able to do it without wrecking the server.

They gave me the go-ahead since we still couldn't contact either admin, and the call queue was flooded with people complaining. I printed out the instructions from the patch, downloaded it to the mail server, and applied it as close to the instructions as I could manage. Then, I started the software. It appeared to run, and testing showed that indeed, mail was back up.

I was a hero. At least until the next day when the main admin got back. Then my root access was taken away. Jerk.

Sometimes, we're called upon to extend beyond our zone of comfort. To do things that are beyond our skill levels, and to perform heroics under dire circumstances. These are things that make us better admins. Learning to deal with the kind of pressure that 20,000 people's programs aren't working and it's up to you, or that electricity is down and $18 billion dollars worth of financial reports aren't getting published and only you can fix it. Maybe it's that your biggest (or only) client had a catastrophe and you're the one handed the shovel. Whatever it is, it's alright to think of yourself as a hero.

Because that's what you are.