Where to put your system monitoring

I'm getting ready to implement my new Nagios monitoring system, and I've been researching best practices.

My current setup is that I have 3 “data sites”, which I consider to be physical locations where servers are kept. The primary site, the backup site, and the soon-to-be-primary site. When the new site becomes primary, the current primary will become backup, and the backup site will go away. Here's how they're setup:



They are geographically diverse. and as you can see, there is limited bandwidth between them.

Nagios is currently set up at the Backup site, and has remained unchanged for the most part since the backup site was the primary (and only) data site. This is not ideal, for a bunch of reasons.

Because of the way Nagios queries things, it is at the mercy of the networking devices between it and the target. If the router in-between goes down, then Nagios sees everything beyond that router as down. You can alleviate the most annoying side effect (dozens or hundreds of alerts) by assigning things beyond the router to be "children" of the router, in which case Nagios will only let you know that the parent is unavailable.

Aside from not having status checking on entire segments of our network in the event of an outage, what if the segment with no network access hosts your mail server? I've had this happen before, and it's disturbing to suddenly receive 2 hours worth of 'down' notifications at 3am. Not a good thing.

To circumvent this type of behavior, I'm going to be employing one nagios at each location:


In the event that one of my sites loses network access, I've still got another host to send messages.

If you monitor, how do you guys arrange your monitoring? If you don't, any plans to start?

Encrypted Filesystems out of the box on CentOS

Like many people that have multiple locations, I sometimes have to get in my car and sneaker-net a hard drive to another facility. Sometimes I ship them via FedEx. In any event, whenever I take a hard drive out of my business, I run the risk of becoming another statistic. These days, it seems that a month doesn't pass where some high profile data has been breached. It happens frequently enough that there's a blog devoted to it.

Anyway, I've been looking for ways to encrypt the drives I transport. It looks like the "best" way is to use TrueCrypt for encrypting the entire device. It's cross platform (Windows, MacOS, and Linux) and has a great interface and is pretty easy to script.

My problem is that it is a comparative pain in the butt to get running on my platform of choice (CentOS/RHEL5). If you look, the only supported Linux versions are Ubuntu and SLES. Yes, I can compile from the source, and I have to test things, but I don't want to have to manually recompile things on production servers. I suppose I could compile it once and package an RPM if I had the time and knowledge (and the time to acquire the knowledge). Instead, I decided that it wasn't the solution for me, unless it was the only solution available. So I kept searching.

Today I chanced upon what I think is a great solution. Using the dm-crypt software along with built in loop devices, it's possible to encrypt a device without using any non-native software.

In the (hopefully) unlikely event that the link I pointed to goes away, here is the (much abridged) process:

If you're using a file, rather than a device (to have an encrypted volume sitting on an otherwise unencrypted filesystem), create the file, here using 'dd':

dd of=/path/to/secretfs bs=1G count=0 seek=8

Setup the loop to point to your file/device:
losetup /dev/loop0 /path/to/secretfs

Create the encrypted volume with cryptsetup:
cryptsetup -y create secretfs /dev/loop0

Create the filesystem on the device:
mkfs.ext3 /dev/mapper/secretfs

Mount the encrypted filesystem:
mount /dev/mapper/secretfs /mnt/cryptofs/secretfs

And now you have access.

To remove the filesyste, perform the last few steps in reverse:
umount /mnt/cryptofs/secretfs
cryptsetup remove secretfs
losetup -d /dev/loop0


Whenever you want to remount the device, just follow all the steps above that don't use dd or create filesystems.

There you go, an easy way to have encrypted volumes on your CentOS/RHEL machines.

Best. Bug. Ever.

The other day we talked about mobile devices for administrators. Today I read about a particularly amazing bug on the Google Android platform that you might be interested seeing.

According to that article, certain firmwares (1.0 TC4-RC29 and below) spawn a hidden terminal window that accepts all keystrokes as commands. *ALL* keystrokes.

The person who discovered the bug was in the middle of explaining to his girlfriend why he had to reboot his phone when his phone rebooted again. Because he typed the word reboot. Good thing he wasn't explaining the various uses of the 'rm' command.

Now THAT is a bug. By the way, the best workaround (aside from updating the firmware) is to edit init.rc and take out the lines at the end that spawn the shell.

Justify the extistance of IT (from Slashdot)

If you've ever wondered how to value your time, or justify your hours, there's a lot of input going on at Slashdot.

I've been lucky to escape this particular issue, but I know there are a lot of people who have to constantly fight for what little resources they're given. Maybe this discussion can help someone out there.

The Creative Admin

Curious. Intelligent. Technical. Detail oriented. Stubborn.

Are these words that describe you, or do they describe the traits required for the position you hold? Is there a difference, after a while?

What if, in addition to those, you had other traits?

Imaginative. Creative. Inventive.

Are these traits (and others like them) that you think would be useful in your job? Would being inspired by your creative side yield positive or negative results? As Tom Limoncelli asked the other day, do you really want your electricians getting creative?

Does your position inspire, require, or even permit you to be creative? It might not, but I know that it can. Art isn't just something that lives in a museum, or hangs on a wall. Art is sometimes intrinsic to science (great example), but art can also happen when science is transcended.

Your work can be your creative outlet. There are flickr groups full of examples. Don't get burned out and lose your will to innovate. It might seem like administration is work-by-rote sometimes, but don't lose sight of the bigger picture, and stay creative.

WPA TKIP Cracked

Well, hell.

I caught on Slashdot today that WPA using TKIP has been compromised. At the moment, only communication from the router to the host is vulnerable, but I can't imagine that it will stay that way for long.

I'm really considering moving my wireless APs to the external network, as opposed to the internal access they have now. That would require anyone on wireless to use a VPN, which has superior encryption anyway, I believe.

Any thoughts?

Stupid Unix Tricks (from Slashdot)

I figure everyone that reads this blog knows about Slashdot, but in case you missed it, here's an entire thread of people contributing their Stupid but Useful Unix Tricks. I figure there's enough of Unix out there that we could all learn some more.

The balance of security and usability

You read it everywhere, from all the security analysts. Security is a process, not a goal. As the implementers and administrators of the control mechanisms, we need to be especially cognizant of that concept.

If you're anything like me, you tend to work on things in waves, or spurts. I'll go for a while, concentrating on one thing for as long as it takes to achieve my goal, then move to the next (probably unrelated) task. When it comes to improving the security of a particular segment of the infrastructure, if we tarry too long in one spot, though, we run the risk of becoming a bit too fervorous in our decisions and wind up becoming draconian.

Rather than becoming like Mordac, we need to view ourselves as enablers of technology. There is a balance to be struck, and that's the hard part. The line is sometimes fuzzy between information security and infrastructure usability. Where you draw will depend on the importance of the data you are protecting, and the organization you're a part of.

Where do you draw that line in your organization? Do you get to decide, or are you at the mercy of policy makers who ignore an entire side of the equation?

Scalability in the face of the incoming onslaught

As you may know, today is election day in the United States. If you're a US citizen and registered to vote, go do it.

Because this is a big election, lots of political sites are going to feel the squeeze. To counter this, many of them are beefing up their facilities ahead of time. High Scalability is taking a look at techniques to improve response.

If you're not familar with High Scalability (the site), you should check it out. They frequently link to very interesting studies in reliability and to techniques that very large sites (google, facebook, etc) use to manage and balance load.

Sysadmin Mobile Devices

Manage any network of sufficient complexity, and eventually you'll want to be alerted to when something breaks. I've mentioned this in general, but not all devices are created equal. What should you look for?

In my devices, I need a full qwerty keyboard. I really do, even if I'm only replying to email. I've seen people texting with a number pad, but my brain is hardwired now to querty. Of course, if you were hardwired to Dvorak like some people are, you might feel differently.

By far, the most important service my phone provides me is email. Since we don't have a Blackberry Enterprise Server, I have a rule on my corporate mail that forwards email to my blackberry. It's actually a combination of rules, crafted to get it to work the way I want. Since I subscribe to all manner of lists and newsletters, those things get sent out around the clock. I don't want to be woken up at 3am because someone on the Likewise Open list can't authenticate their AIX machine. For this reason, I have my mail rules setup to forward everything (excluding some high-traffic lists) to the blackberry, and then from 10pm till 8am, only emails originating from our externally-facing domain are forwarded. Since all of my internal cron job notifications are sent from the imaginary domain we use for internal resolution, they don't get forwarded. I have specifically set up Nagios to send emails from an external-domain account, so I get them all the time. This ensures that my bosses can get a hold of me, and that I'm aware of any critical weirdness happening at any hour of the day or night.

Also important to me is an SSH client. I don't make full use of mine yet, for reasons I'll explain, but I can administer my firewalls from outside with my phone. I have heard, and someone please correct me if I'm wrong, that if your corporation has a Blackbery Enterprise Server, you can use that connection to reach internal hosts. I don't know that I'm going to be running my own BBES anytime soon, but that's a strong argument for it. There appear to be lots of remote desktop solutions available too.

All in all, my blackberry provides me with sufficient access to resources. I wish there was a VPN solution for it that I was convinced would work with my Netscreen solution , but I suppose you can't have everything.

Of course, I'm not suggesting that the blackberry is the bees knees, as it were. I'm sure there are better solutions out there. I'd like to think the iPhone would be amazing, but I don't know how typing commands on the keyboard would go. I doubt the auto-correct on spelling would like some of the unix commands I'd be typing.

What do you use for a mobile device? Can you do any remote administration through it, or is it just for communication, and you fall back to your laptop in emergencies?