Saturday, August 30, 2008

A look at undersea cables

Browsing through Digg today, I chanced across this article about Google expanding their undersea cable portfolio. Aside from increasing the inter-Asian bandwidth by over 7Tb/s, it will be Google's 2nd undersea cable, the first being the transPacific cable known as "Unity". As recently as a year ago, the first cable was just a rumor. In February, they confirmed that it would be in place and operational by 2010. The first article I linked to also mentions that Google is looking at an African fiber to replace the aging SAT-3 cable. Offhand, to give you some idea of the magnitude of the new fiber, SAT-3's bandwidth is a paltry 130Gb/s.

What does this have to do with small systems administration? Not much specifically, to be honest, but it's interesting to me. I look at the global fiber network as a maybe a macrocosm of my own network. I look at the global fiber maps and note the critical junctures, the (sometimes lack of) redundant bandwidth in certain corners of the world.

If you're interested in this as well, here are some interesting links that I found. has a couple of pages dedicated to mapping the exact spots where each of the trans-Atlantic and trans-Pacific cable landings. This is a very, very bandwidth intensive page. Far more interesting are the maps at the top of each page that shows the logical circuits. Here, I'll save you time and bandwidth: Atlantic and Pacific.

This map from CNET gives you some idea of the intercontinental bandwidth available around the globe.

Here is a series of maps showing the history of undersea cables.

If you're curious as to how many cables are out there, Wikipedia has a page dedicated to listing all the cables in the individual bodies of water. The main page on submarine communications cables has a great schematic of the composition of undersea cables. How cool does this look:

Thursday, August 28, 2008

Password retention and storage

I got an email from a reader yesterday asking about how I generated and stored my passwords securely. The reader was interested in what methods were available to sysadmins for managing diverse passwords for different machines and devices.

I had to laugh at my password generation scheme (run 'fortune' a couple of times, pick some random words and throw a random character between them), and my password storage is nothing to brag about either.

What methods do you use in your infrastructure to generate / store passwords?

When is a sysadmin not a sysadmin?

I really like my job. I love connecting complex bits of hardware into a working infrastructure. Taking inanimate bits and making a living, breathing network out of them is exciting. It's the best model train set you could ask for.

There are also parts of my job that I intensely dislike. I'm not talking about the "just sort of unpleasant" tasks like user support. I don't actually mind helping people out. Part of my job is making sure other people can do theirs, and as such, I don't mind lending a hand.

What I don't like are generic administrative tasks that only take time away from the rest of my duties. Here, I'm talking about things like changing tapes, keeping track of users' software licenses, and more to the point, dealing with billing details for network contracts.

I received a voicemail today, forwarded to me by someone in the corporate office regarding our Level3 account. When we moved out of the downtown NYC office months ago, and we cancelled our Level3 account, but from everything I can figure, they didn't cancel our Level3 account. Now I've got to try to get a person on the phone from Level3 to figure out what is going on. I'm not worried that we actually owe them anything, because I've got the email from them with the disconnect order, but the rigamarole in making them understand that isn't shaping up very well. Last night at 5:00pm, I sat on hold for 45 minutes before I gave up and went home. I'll try again this morning, but I'm not anticipating a good experience.

Let me advise you this, to stay away from Level3 for your dedicated circuits, because getting any kind of help from them is next to impossible, and the service wasn't even great to begin with. They are the only bandwidth provider I've ever have who gave me a hard time about trying to contact an internal engineer. I needed to ask questions about the MLPPP-bonded T1s I was going to try, and it took 2 months to get their guy to call me back. To make matters more fun, every single contact there in my contact list has fled, so I don't even have a real person's name or number who works there. Stay away.

Dedicated ESXi: Server or something else?

Christopher Hoff, over at Rational Security discussed an interesting idea last week about VMWare's ESXi. One of his readers theorized that maybe ESXi-based servers shouldn't be considered as servers in some cases, but more like quasi-blade-chassis, since they essentially provide the same functionality.

They're exploiting the idea for auditing, but since I'm really too small for that sort of thing right now, I don't have to worry about that, much. I also don't worry about ESXi much, since even though it's free, the Virtual Infrastructure to really use it costs thousands of dollars.

I just figured I'd throw this out there to see what you all thought of it.

[Update] Fixed broken link. Thanks DDressler!

Wednesday, August 27, 2008

BGP Security issue

If you are running a lot of routers, with BGP peering, then you hopefully already know about this. If you don't use BGP, then there's not really much you can do besides plead to your upstream provider, but regardless, you probably should know.

Security researchers have revealed that there is a major flaw in BGP that allows an attacker in an arbitrary location to redirect BGP traffic to their site and then reroute it to it's intended destination.

Of course, encrypted streams would make this a non-issue, but think about all the unencrypted traffic on the internet. That's a lot of information that you don't want someone getting their hands on.

New Sysadmin Podcast Series

Ben Rockwood is coming out with a new Sysadmin-centric podcast soon. Judging from his blog entries, it's going to be really informative.

Read the announcement here:

You might have to copy and paste that url in. Ben seems to be blocking my links? I'm not sure what the deal is with that.

Now if I could just comment on his blog entries...can someone who is able to throw a trackback on there for me?

NetworkWorld features one of us

I feel a special kind of kinship with other people who are in a similar boat as myself. That's actually why I started this blog, to work with other "standalone sysadmins", people doing work that would normally require the skillsets of several individuals in larger companies.

Imagine my surprise when I opened my NetworkWorld magazine and saw this feature about Justin King, a standalone sysadmin if there ever was one.

Justin manages a total of 130 nodes in the Neuro imaging lab at Baylor College of Medicine. He's got almost 120TB of storage, a compute cluster, and normal servers to worry about. The one break that Justin has is that his user base is technically oriented and doesn't need hand holding.

Read the article, it covers the struggle that us single-handed sysadmins have to deal with, and it's nice to see one of us get some recognition.

10 professional skills that no IT administrator should be without

Whether you're a network administrator, systems administrator, cable monkey, or general IT grunt, there are a few professional skills that are indispensable in order to prosper. Sure, you can survive being surly and bitter, but to reach your full potential, it would be wise to take this list into consideration.

10) When something is broke, fix it
This can be as simple as a loose cable end, or as complex as an inefficient process. It's not your bailiwick to fix every problem in the entire infrastructure (then again, maybe it is), but an eye for improving things is highly appreciated in the right sort of company. If you see a way to improve something, talk to the appropriate people to see what can be done. You are part of the company, and by improving the whole, you're making things better for yourself, too.

9) Manage your time efficiently
Tom Limoncelli's book "Time Management for System Administrators" goes far more into detail than I can here, and the book is not just for sysadmins, despite the title. To summarize an important point, keep track of what you have to do, and when you have to do it. It's either due right now, it's due soon, or it's due later. If you categorize by those identifiers, you'll never have something sneak up on you.

8) Learn from people who know more than you
Discovery of new things on your own is exciting. You're full of wonder when you chance upon an unfamiliar subject. It's no doubt similar to the feeling Magellan or Columbus had. Like those people, you too are probably discovering something that other people have known about for a long, long time. While you should never stop exploring on your own, never be too proud to learn from the people who have been there before you. Even the guy who invented calculus stood on the shoulders of giants. You might be able to borrow someone's stepladder, anyway.

7) Organization shall set you free
There are people who have to keep track of lots and lots of things. Some of those people are librarians, and they use something called a Dewey Decimal System to manage thousands of discrete items. If you've ever wandered through a library just glancing at shelves, you've probably been amazed by how bizarre some of the book groupings can be. Despite the seemingly random categories, any librarian worth their salt can tell you where a book is filed in a moment's notice. That's because, despite what you or I might think of their cataloging system, it works. I'm certainly not suggesting the Dewey Decimal System for your network documents / asset tracking, but it's important that you have a system that you know how to use, and that it's one which works for you. A pile of papers isn't a system, and neither is throwing everything in the root of your home directory. Make it easy to find data that you need, and use the tools at your command to make it happen.

6) Pick your fights
You probably work with at least one other person. If that person is in IT, you'll probably disagree as to the best course of action. If that person is not in IT, it's almost certain that you'll disagree about the best course of action. As IT workers, we're exposed to a large amount of data, some of which is so obscure or esoteric that I'm tempted to call it trivia, however the lump sum of that data, when added to our experience becomes knowledge, and we use this knowledge to make judgments about procedures that other people might think are arbitrary decisions. Example: It's a bad idea to run CAT5 along the power conduit that feeds the florescent lights. It just is. We could explain why, but the person who wanted to do it wouldn't care why we refused. They just know that they aren't going to get their network cable. Discussions of this nature will be more or less serious, depending on the day and the phase of the moon. When it comes to the less serious discussions, my advice to you is identical to the advice that someone very wise gave me about my then-upcoming wedding. When the two of you disagree, go along with whoever cares more. If it's a bad idea, and the person knows that, but really really wants it, maybe you should consider giving it to them. Maybe.

5) Competence is next to Godliness
I was at Chipotle the other day, and I saw a 12 year old Honda with a completely useless rear spoiler that rose a foot above roof height. My initial thought was "I bet that guy doesn't race his car". Why not? Because he's just showing off; there's no substance behind his style. You can talk as big as you want about your skills, but unless they're real, it's all hot air. Always seek to increase your competency rather than people's opinion of you. If you succeed in the first, the second will follow.

4) Be honest...enough
Ah, honesty; you wear so many guises. There are laws against certain kinds of dishonesty. There are societal mores that frown upon others. Then, there is a certain kind of etiquette that requires a bit of fudging the truth. It's important to realize that people sometimes need to be let down gently, particularly when they're proud of something they did, or think they did, when in fact, they've screwed the pooch, so to speak. Honesty is also important when something breaks, though it's probably more important to know that the finger you use to point to the cause would be better used to fix the problem.

3) Take responsibility
Just as important as not pointing fingers at other people is recognizing when, through some fault of your own, something has gone to hell and someone else noticed. I've honestly never seen a member of management more amazed than when someone stood up, looked them in the eye, and took full responsibility for the problem. Worming your way out of a jam is a time honored tradition, and taking the opposite stance will confuse and confound those to whom you plead. An honest apology coupled with an assurance of future improvement, and perhaps a quick overview of that path, is the surest way to gain respect from people who are used to seeing the worst of people.

2) Be a team player
Sometimes, on occasion, and I must emphasize the rarity of this event, it is not all about you. In this instance, you might be part of a multi-department ninja-strike-team, or maybe you've been assigned a partner to get a major issue taken care of, but whatever the particulars, in dire times such as these, someone else is involved in the task at hand. When you inevitably succeed (is there another option?), to the victors go the spoils. In this case, it might be a raise, or more likely, a congratulatory email. The proper reaction is, despite how weasely your teammate(s) might have been, to share the responsibility of success. In fact, I've found that a good rule of thumb is to never assume more responsibility for success than the amount of blame you would have ascribed to your partner(s) if the project had failed.

1) Have an exit strategy
It's sad, but not all of our endeavors end in unblemished success. In fact, some times, they just up and fail. We're talking real end-in-fire type stuff here. You can usually tell when something like this is going to happen, because you find yourself pushing towards a solution that shows no sign of yielding to your will. Lots of times, you'll make little bits of progress as large sections of time leap by. You approach the deadline with no real progress to show, and your choice is to present a half (if you're lucky) finished product, or hack together some kludge that doesn't pass muster and hope you can fix it later while no one notices. Neither of these options is particularly appealing, so it's important to set a limit for yourself. If it's not 80% by this date (where "this date" is a decent amount of time ahead of the deadline) then we're going with Plan B. Of course, this necessitates having a Plan B, but I think that's a good idea regardless of what activities you're engaged in.

I certainly didn't cover everything you need, but I got some of the big ones, I think. If you disagree, or have more to add, please comment! If you liked the list, feel free to Digg it. If you're tired of these stupid lists of things, drop me a line at and let me know.

Tuesday, August 26, 2008

Discussion of SSL Web Certs

There's some excellent discussion going on over at TechRepublic regarding SSL Certificates, and the way they're treated by modern browsers.

Firefox 3 all but refuses to let you access a site with an expired or self-signed certificate. Lots of other browsers are becoming more and more militant about it.

The author of that article, Michael Kassner, brings up a good point regarding Extended Validation certificates, which I didn't even know about. The discussion in that thread was dealing with whether SSL certificates are useful solely for identification purposes or if they have an alternate use in sites that only wish to have encrypted traffic streams.

Read the article and join in on the conversation. I'm interested in learning more about people's opinions on the subject.

Display System DNS on OSX

This is mostly a note for myself, but someone else might find it useful.

To display the current DNS resolvers on OS X, use "scutil --dns"

5 Ways to improve your network without breaking the bank

In reality, there are far more than 5 ways to easily improve your network's security and reliability, without spending hardly any money at all. I've just briefly gone over the most blatant 5 that make the biggest difference for next to no expenditure. Sure, a few office supplies might need to be sacrificed for the good of the many, and you will have to change the way you approach some problems, but read this list with an open mind. You might find it more efficient than the things you've been doing.

5) Take care of your networking cables
Nothing will kill packets with more random violence than an ethernet cable that hasn't been taken care of. Whether the culprit is a loose end, bent wires from folding it back on itself, or accidentally cutting it when you were removing plastic zip ties, the end result is that your quality degrades and packets get dropped. This can be prevented by recrimping the ends when necessary, or replacing the whole cable if need be.

4) Set your wireless power level
You probably don't work in an area that covers 100,000 square feet of space, which means that lots of the wireless signal you're sending out is being wasted. More troubling, it's much easier for the casual war driver to pick them up from the parking lot (or adjacent floors). There's just no need. Most APs support adjusting the power output from the radio. Lower the signal until you can't get it in the farthest reaches of your office, then bump it back up until you can get it again. It won't defeat someone dedicated who uses a pringles can, but it's better than nothing. Also, make sure you use WPA.

3) Optimize DNS
If I were to wildly make up numbers, I'd say that 60% of weird network errors can actually be traced back to DNS, particularly if your infrastructure has multiple DNS servers which are all manually edited then updated. It's easy to fatfinger a line in the config file, or forget to update all the servers but one, or to leave out a reverse DNS entry for a host, and with the right (or wrong) host configurations, each of these can cause seemingly random havoc on unsuspecting users. "Why is it taking to long to ssh into BoxA?". It might be a network issue, it could be a server problem, or, in my case, the reverse DNS entry COULD have an extra : in it. Solve these kinds of problems by scripting DNS changes. Do less by hand and eliminate accidental screw ups. That way you can concentrate on all the bad things which you do on purpose. This entire process of automating changes and then documenting them is called "Change Management". Look into it, it's a Good Thing(tm)

2) Monitor Bandwidth
If you've got managed switches, there's no excuse to be wasting those expensive configuration and reporting abilities. Pretty much every networking device and almost every network host has the ability to be monitored using SNMP. To ignore this potential source of pretty graphs is foolish. Get MRTG, or even better, Cacti, and get to work. Knowledge is power, and learning that what should be a small filesync is actually backing up a user's entire iPod directory, every night, helps you in many ways. Learning discretion by not hitting them with a foam bat, for one.

1) For the love of God, document!
If it's something that you could concievibly have to do again, and it took you longer than 10 seconds to figure out, document it. Document it someplace that makes sense. Put enough detail in that you can recreate your actions, or better yet, someone else can recreate your actions. The last thing you want is to get a promotion to a corner office and then have some peon come bothering you about network configurations while you should be lighting cigars with $100 bills. Document properly and this won't happen. I promise.

Thanks for listening. There are many, many simple ways of improving your network. Post some of your favorites below. If you liked the article, make sure to Digg it

Update on the VPN Issue

How appropriate was yesterday's blog entry? After writing it, I left for work. On my way to work, I got a call about the VPN not working for one of my users.

I spent probably two hours troubleshooting her previously-working IPsec tunnel to no avail. I finally gave up and told her I had to get off the phone to think about it. After some contemplation, I decided that I'd go for the gold and break out the new SSL VPN a bit early.

Of course, the only problem was that A) I had never touched one before, and B) The documentation was pretty meagre.

I hoped for the best as I pulled the device out of the box, then slipped the plastic bag from the chassis. It is a testament to how easy to use that netgear is that I had my user up on the SSL VPN and working within 2 hours. And it worked the first time I tried it with her.

If there's any kind of interest in a formal review of the Netgear SSL VPN device, I'll be happy to type it out, but in a nutshell, if you're dealing with a small office of users, and you want a firewall with VPN capabilities, that is the one to get.

Monday, August 25, 2008

VPN Woes

Have I told you about my VPN problems? No? Well, sit down a spell and have a listen.

When it comes to my company, we've got two types of VPNs, really. There are the site-to-site VPNs, which connect, well, sites. My office's router (a cluster of Juniper Netscreen 5GTs) have VPNs set up to each of the other sites. It's sort of a mesh configuration, since every site has every other site connected via VPN policy, but with only a few locations, this isn't too unbearable. I'd rather have an MPLS network, but hey, I take what I can get.

The real problem becomes user VPNs. See, we've got a primary site and a secondary site, and something like 15 users who each need to be able to connect to both locations. This means that I've got to maintain 30 accounts on the firewalls, AND 15 user machines which connect up. Neither is fun, but the user machines are the worst part.

We use Juniper Netscreens for the VPNs, and our Mac users typically use VPN Tracker or IPSecuritas. VPN Tracker is easy to set up , and commercial. IPSecuritas is free, but much harder to configure. Both do work, however, which makes them better off than our Windows users. Our Windows users are burdened with Netscreen Remote, and an old version, at that. It's just generally bad. It gets confused a lot, and requires reboots to clear the IP configuration so that traffic actually reaches the VPN. Sometimes it will die a slow death; the other day I had a user who could connect to most of the resources on the VPN...then they could only connect to a couple. By the end, the only thing they could reach was the jabber server, over which they were talking to me. A reboot fixed the problem, of course. Lots of times, we'll have people who can get email, can ping everything, but can't SSH into anything.

To fix these strange, strange issues, I'm trying another solution: an SSL vpn.

You might know that IPSec operates over UDP port 500, and requires installed software to be configured beforehand. Basically, an SSL vpn differs from an IPSec VPN by transmitting the traffic over encrypted web-traffic, to port 443 on the VPN device. This allows the client to connect to the VPN merely by visiting a webpage and authenticating themselves. At that point, a java or activeX program is downloaded and installed which acts as a pre-configured VPN client which transmits internal-destined traffic over the SSL tunnel. Anyone who tells you this is a "clientless" operation is lying. The client is just downloaded on the fly.

Anyway, the device I'm going to be using is the Netgear Dual Wan Gigabit SSL VPN. I honestly have no idea if it will work or not, but I'll be sure to let you know.

I'll probably be testing it later this week, so the update on it should come next week.

Thursday, August 21, 2008

Burnout and the toll it takes

Jack Hughes, over at the Tech Teapot, mentions a very appropriate subject for too many systems administrators: burnout.

As sysadmins, we're nearly always the go-to person for whatever happens. After a while, we start to get used to it, and lots of times, we can develop a hero complex, carrying the weight of the world on our shoulders, at least in our minds. This isn't healthy for a lot of reasons, the most important of which is your health.

Here's an example of what taking your job too seriously can do to you:

Part One

Part Two

Not to ruin the ending, but the most disgusting part is that, while the guy was taking medical leave, his company fired him. To be completely honest, he's much better off without a company like that, and if your company would do the same thing, then so are you.

To quote Peter Gibbons, "We don't have a lot of time on this earth. We weren't meant to spend it this way. Human beings were not meant to sit in little cubicles staring at computer screens all day..."

Even one of the most preeminent Systems Administrators around, Tom Limoncelli advocates leaving the pressure at work when you head home. For those of us on call 24/7/365, that can be a little hard, but it's important to try.

Wednesday, August 20, 2008

Really bad cabling jobs!

I found this amusing/depressing series of pages involving some epic wiring jobs

God help you if you're stuck with anything remotely this bad.

Creative Community Hacking Forum

This is a cool idea.

"JustHackIt". Share creative ideas with other creative people. I like it.

Project Management Software

My company is in the market for some online collaborative project management software. Since I have never used any of it, I'm appealing to you, the reader. What do you use (if anything) for this? 

I've used Microsoft Project a very small amount, and the open source solution Planner a bit more, but neither are web based or collaborative.

A quick search on Freshmeat returns a lot of likely candidates, but I think we're looking for something commercial, for the supported aspect. On the down side, this will cost the company a certain amount of money every month. On the up side, I think they're finally realizing that my time is finite.

Here are the options we're looking at:



Liquid Planner




Anyone have any experience with one (or more) of these, or any others? I'd be very interested to hear your thoughts.

Tuesday, August 19, 2008

Wireless Security (of lack thereof)

TechRepublic's Network Administrator blog has an interesting entry today regarding wireless security, which is a topic certainly bears some thinking about.

This weekend, I was actually doing some research on wifi security this weekend with my wife's laptop. My powerbook's airport extreme card isn't so great for cracking networks, but my wife's Dell has an intel chip that will work for most parts, though packet injection isn't supported.

Anyway, I've figured out that WPA-PSK is alright for me, especially at home. As long as I use a good enough key, the amount of time spent cracking it would be in excess of the reward of stealing my wifi, especially when you consider that one of my neighbors is using WAP.

On the other hand, there are some improvements that I'm going to set up for work. The first of which is, when I finally get everyone connected to the active directory infrastructure I'm building, I'll authenticate the wireless against a radius server. The AP we're using supports that, thankfully.

One of the links that the TechRepublic blog entry links to is this one: how to prevent parking lot attacks. This is an interesting read, since it involves things like radiowave-reducing paint and window covers which block 2.4ghz radio waves. Interesting stuff, most of which can be implemented for $0 by reducing the strength of radio broadcasts on your AP. Nevertheless, if I suddenly find myself with oodles of money and the need to protect a server room, I'll use the advice.

Monday, August 18, 2008

Introduction to RAID levels

If you've never worked with large systems with a bunch of disks, you probably don't have a lot of experience with RAID. RAID stands for Redundant Array of Inexpensive Disks (or if you ask some people, the I stands for Independent). has a good article on how the most common RAID levels work.

For a much more in-depth discussion (and visualizations of the way data gets written to where), check out this article from

It all boils down to how important the information is, how much disk space you need, and how many drives you have at your disposal.

Friday, August 15, 2008

OpenOffice trick to auto-increment IP addresses

I've been fleshing out my IP address spreadsheets, and I've been using a trick that I didn't know about for a long time, till someone on a forum showed me. It was indispensable.

You know how, if you want a series of numbers down a row, you can click the lower right hand corner, and drag it down, and it autoincrements the numbers for you? Well, in oocalc, if you try that with IP addresses, it just copies the original IP address. I don't know why, but it does. MS Office gets it right, heck I think even K Office will do it. OpenOffice, not so much.

The trick to doing it is to put a space in front of the IP address, first.
"" will allow you to drag it down as far as you want, incrementing the last octet.

Most sought after skills in IT

I know I said I'd only post HOWTOs on Fridays, but the one I'm working on isn't done, and I saw this and had to wonder how accurate it was:

My company admittedly isn't hiring for any of those. The only opening we have right now is for a Java developer, and they're not even looking for that yet. We're not a Microsoft shop, so that takes away 3 from the list right away, and my company is nowhere near big enough to benefit from an Enterprise Resource Planner (ERP) like SAP.

Just thought I'd mention this here for you to ponder. Anyone hiring for these positions?

Thursday, August 14, 2008

Remote Management and its Limits

Well, it appears that one of the cables in my new rack is bad. Specifically, it looks like it's the one going to the "red" switch. It's not really surprising, I suppose, given the last minute rush to find color matched cables. What this means is that I've got to walk someone through replacing it remotely.

Having remote management ability is important. Whether it's IP KVMs (we've got that), or the ability to perform web/ssh/telnet/rdp (we've got all those too), there's literally only so much you can do when you're not at the site, and that's frustrating. I'm 400 miles away; I can't just hop on over and change out a cable, and the timetable for us getting our stuff turned on precludes anything except using the remote hands service of the colocation.

I know most of us have a single location for our IT stuff, and it's usually in the same building we do work in. Heck, at the ISP, my office was /in/ the server room. Bad for the ears, but I could hear when things like hard drives were dying. Pinpointing an exact location was a little harder, since almost none of the hard drives were hot-swappable.

Anyway, at this point, I'm extremely glad we have support at the colocation. If you decide to move into a colo, I highly recommend getting one with remote hands.

Wednesday, August 13, 2008

Recycling your e-waste

My central Ohio office is closing very shortly. In the process of closing, we're going to be getting rid of quite a lot of servers, old generic machines, and other miscellaneous hardware.

I worked with a recycler last time I had extra hardware, but I'm not going to post the name of the company, because I was really unimpressed with them. I asked for certificates of destruction for over a year before they stopped responding to my emails.

This time, I'm looking at going other routes. The top of the list so far is FreeGeek, a not-for-profit organization that takes older hardware, refurbishes it, and donates it to local nonprofit organizations. There are a few local chapters, including the one in Columbus, OH. They guarantee that hard drives are wiped, and they'll take any equipment, regardless of whether it works.

I can't say I have a lot of free time, but if I did, and I was going to be around more often, I'd love to volunteer there. I don't see a NJ chapter, so maybe there's a need up there where I'm headed.

If there's not one around you, there is probably another worthy cause for you to volunteer for in your area. See this April article from LinuxJournal for more options.

Give the gift of knowledge to people who know less than you, and for once, I don't mean your bosses. ;-)

Tuesday, August 12, 2008

Email to the position, not the person

Just a small note that comes to mind as I register this Belkin IP-KVM that just came across my desk.

When you go to register an asset, hardware or software, invariably you are asked for an email address. The initial knee jerk reaction is to use your normal email address, but they really don't want your email address, they want the email address of the systems administrator. Sure, you're the systems administrator, but that won't matter later, when you're not.

I use an alias on my normal user account, for these occasions. I know that I'm not going to be in this job for another 30 years (at least I hope not, Dear God), and when I hand the reins to someone else, they're not going to want to deal with trying to figure out whether something is registered in my name, or the guy who came before me, or if they already did it and forgot. So I use my sysadmin alias, and they will too, I imagine.

Just another courtesy you can extend to your future usurpers.

Simple and obvious first

Logically, when you're trying to debug an issue, you should start at the bottom. The simple answers are the easiest and the quickest, and let you get back to solving the real problems. Unfortunately, something usually comes between you and the easy answers: assumptions.

In our ever-present quest to go faster and get through things quicker, we cut corners, we make dumb mistakes, and we take on faith that others have performed due diligence so that we can start with high-level troubleshooting.

Having been a tech support agent for years, I'm acutely aware of this, and having been a sysadmin for over half a decade, you would think that I would be used to this situation, but that's not the case. Oh, how I wish it were.

The reason that I bring this up is that yesterday I spent a couple of hours troubleshooting a missing set of quotes. I know, I know, how can you spend a couple of hours fixing something that obvious? It started with a simple issue: A shell script wasn't passing the right filename to a program that it called.

I was brought in to figure out a way to get an argument to the script passed to the program correctly. I was given the example:

$ arg1 arg2 "/path/to/file with spaces in name"

Inside, a program was called like this:

/path/to/program $1 $2 $3

although I didn't know that at the time. I solved the first problem, that the argument to the shell script wasn't being interpreted correctly.

I eventually wrote a slick for loop to handle the arguments without breaking the filename into separate tokens. After testing it, I presented it to the person who asked me to look at it. After installing it into the program, it still didn't work.

By this time, he was tired of dealing with it and asked me to work on the actual script. Of course, I looked at the script, saw the above line of code, and added quotes around the $3.

Obviously, the script worked perfectly after that.

My mistake was to assume that the script was written right and that the real issue was the one I was given. It might have taken 30 extra seconds to look at the contents of the script in the beginning. But I didn't, and lost a couple of hours for my trouble.

Let this be a lesson to me. And you. Learn from my example, and verify your assumptions

Monday, August 11, 2008

Disk Quotas

Strangely enough, for someone who complains so much about storage space, I've never setup disk quotas in Linux.

I guess it's because I've never had a large installed user base in Linux, and didn't feel the need to provide hard limits for the users. I would occasionally run "du -h -s *" in the volume root and send a reminder to people with unnecessarily large directories, but never any mandatory requirement.

I believe that's going to change. I've been thinking about it, and since our space is finite, there's no reason that the user home directories should be allowed to inexorably fill up the entire partition. To make sure it doesn't happen, I'll use disk quotas.

Nikesh Jauhari over at Linux Poison wrote a quick little introduction to disk quotas in Linux. It's about the simplest explanation I've seen, and I like simple. For the Windows reliant, Microsoft published an article entitled Managing Disk Quotas in Windows Server 2003 and Windows XP, which should cover most of the installs out there.

Anyone else been looking at this for a while? And while I'm asking, how much storage is enough storage for your users?

Friday, August 8, 2008

Sources of wisdom and experience

If you're a sysadmin (and given the fact that you're reading this, you're at least interested), you may be familiar with SAGE, which I've mentioned before. Every year, SAGE and USENIX put on a convention called LISA, the Large Installation Systems Administration conference.

Each year, there are plenty of speakers and programs covering topics of interest to all sorts of programmers and administrators. After the conference, the presentations are put online for the USENIX/SAGE members. After a year, they're released to the general public.

2007's presentations have recently been made available, and all the previous years' are still online.

Even though the presentations are a year old, they're still very educational, and an excellent way to pack in something interesting to your schedule.

Thursday, August 7, 2008

Linux authentication against Active Directory

If you've been reading the blog for a while, you might remember me saying that I have been (and perpetually am) fighting with centralized authentication. Well, I'm here to update you.

I have found the answer, at least for authentication against Active Directory. Salvation, thy name is Likewise Software

Likewise produces two pieces of software. The first is Likewise Open, a free piece of software that authenticates your Linux/Mac/AIX/etc machine against active directory. It does this by making several changes to the default configuration of things like PAM and Samba. The end result is that you can log into your linux machine with Windows' Active Directory credentials. It's very neat, it's free, it's incredibly easy to install AND uninstall. Best of all, it really really integrates with Active Directory in, as far as I can tell given what little I know of AD, the Right Way(tm). I submit for your approval:

(click to embiggen)

You can see there, all of the machines that I've installed this on show up in Active Directory. When I log into the machine, I can log in with domain credentials and it knows about my default group (as specified in AD Users and Computers):

bandman@newcastle[504]:~$ ssh int\\msimmons@a-fs1
Last login: Thu Aug 7 10:18:02 2008 from
[INT\msimmons@a-fs1 ~]$ ls -al
total 36
drwxr-xr-x 3 INT\msimmons INT\enterprise^admins 4096 Aug 7 10:18 .
drwxr-xr-x 3 root root 4096 Aug 5 21:17 ..
-rw------- 1 INT\msimmons INT\enterprise^admins 124 Aug 7 10:18 .bash_history
-rw-r--r-- 1 INT\msimmons INT\enterprise^admins 33 Aug 5 21:17 .bash_logout
-rw-r--r-- 1 INT\msimmons INT\enterprise^admins 176 Aug 5 21:17 .bash_profile
-rw-r--r-- 1 INT\msimmons INT\enterprise^admins 124 Aug 5 21:17 .bashrc
-rw-r--r-- 1 INT\msimmons INT\enterprise^admins 32 Aug 5 21:17 .k5login
drwxr-xr-x 4 INT\msimmons INT\enterprise^admins 4096 Aug 5 21:17 .mozilla
-rw------- 1 INT\msimmons INT\enterprise^admins 58 Aug 7 10:18 .Xauthority

Let me tell you, I'm impressed.

Now, this is just Likewise Open, the free version. It only modifies the configuration on the Unix based machines. Also available is Likewise Enterprise, which provides the same service, but goes above and beyond Likewise Open, in that it actually makes changes to the AD structure. As far as I know, all of those changes are benign, in that they break nothing related to any other Windows service. I haven't worked my way through the nearly 500 pages of documentation that I had printed and bound at Kinkos the other day.

I'm sure this post sounded like a commercial, but it's not. I haven't been paid (or even contacted, other than the initial autogen email) by Likewise software, I'm just a grateful user who is happy to share knowledge of a tool that works. Finally.

Wednesday, August 6, 2008

Just because it's strange

Speed Cabling.

Desktop Workspaces

This is a random post, but I've been thinking about desktops.

I've never really been a harsh critic of the Windows desktop idea. It seems pretty straight forward, and like it would be easy to use for people who aren't used to computers. That is, until Vista, but there aren't so many people now who have never used a computer. Maybe the need for simplicity is gone in the Windows environment. Either way, it always seemed to work, if a little slowly in some cases. After I got used to the various X Windows managers, though, Windows definitely seemed far more rigid, in terms of configurability, and completely hampered by only having one desktop, as opposed to multiple.

One thing was that it always seemed to be getting in my way. I could hide the task bar, and use keyboard shortcuts, but I never got the shortcuts set up the way I wanted. Sure, I could download some shareware programs to setup the keyboard right and launch programs that I wanted with the keystrokes I wanted, but it stopped working unless I paid $20, and anyway, it seemed like a hack.

When I got started in Linux, I tried KDE, and it was pretty ugly at that time (1997ish). I tried FVWM, and I used that for a while, or it's modified FVWM95, but they wore on me pretty quickly. It wasn't until I tried WindowMaker that I found my groove.

Fast forward to now. With the exception of the background image, I have used the same desktop environment for essentially 7 years. The exact same keyboard shortcuts and window preferences have followed me around (thanks to everything conveniently living in ~/GNUstep). I even carried my exact environment from Slackware when I switched to Ubuntu.

I've got pretty heavily customized Gnome and KDE installs on my Ubuntu box, but I never log into them unles there's a very specific (and rare reason). The rest of the time, I'm in WindowMaker, and the reason is that, after 7 years, I'm blindingly fast.

I can open terminal windows, type commands, and get the responses faster than someone in Windows could click on the putty icon. I've got WindowMaker setup to dynamically create desktops, so if I need a new working space, I just scroll past the last existing one. It consumes nearly no resources, and it gives amazing amount of desktop space.

Overall, if I was just running a desktop that I expected to be doing useful things locally on, I'd probably run KDE, but as an admin who constantly needs to ssh into remote hosts to get things done, I can't recommend WindowMaker enough.

I spent some time this past weekend working on my laptop at home. I've mentioned before how much fun it is, which is to say, very little. This time was different, though. I brought a 19" LCD from the office, because I knew I'd be working quite a bit, and I've got to say, it made a huge difference. I almost liked it. I felt like I was being productive again.

In fact, the only down side was that I was using OS X, which in my opinion is the finest laptop OS available, as a desktop. It was still slow and kludgy compared to my linux machine, but it was far better than trying to do serious work on a small little screen.

Monday, August 4, 2008

Flash Drives: Archival Media?

I have lots of client data to backup. I use tapes for long term storage of data I'm hopefully never going to have to pull off, but if I need to, I can. Our total amount of storage is such that a full backup now takes over 2 days to store, on 3 320GB tapes. It's a lot of data.

There are times, however, when I just want to archive a small bit. For example, I have a 5GB (compressed) chunk of data I'd like to archive and send to the ClientManager. I could send it over the network, but there's a T1 between me and her. That would take a "long time", and it would take up unnecessary space on her computer. I could store it on the fileserver, but that's the space I'm trying to free up.

If it were just a little bit smaller, I'd send her an archival DVD of the data, but 5GB won't fit, and it won't compress any further. There are some very high density optical disks on the horizon, but without spending thousands of dollars, I don't see a solution down that path.

So here's what I'm considering. Write the data to tape, as normal, but in addition, provide the client services manager with the data encrypted on a flash drive. I can use full disk encryption with TrueCrypt, which will work on Linux, Mac, and Windows, and provides for the security of the data. In addition, if the drive breaks or gets lost, we've still got the original tape archive in storage.

Any thoughts?

Sunday, August 3, 2008

Couple of interesting links for your Sunday Enjoyment

I didn't have much better to do right now than browse reddit, digg, and others, and I found a couple of interesting articles that might interest you all.

First, Berkeley's Open Computing Facility is going to be offering a Unix Systems Administrator Class. They offer courses for those new to Unix, and those with pre-existing knowledge. 
I think it would be really interesting to go to those classes. As someone who already works in the role, but never having been educated through "official" channels, it would be neat to see what they teach. 

2nd link is a document from IBM discussing the merits of treating a virtual linux machine as a managed object.

I also know of several people who want to become IT Consultants. That's an intro guide from Career Planet.

That's all for this lazy Sunday. Catch you all tomorrow.

Friday, August 1, 2008

HOWTO: Punch down blocks for in-building wiring

Punch down blocks are used for when you need to run wires long distances, typically between distribution points ( things like the MDF, or Main Distribution Facility, otherwise known as the main telco room on the primary floor of the building), comms closets, and the like. They are used, rather than normal RJ45 jacks, because they are simpler, less prone to breaking, and don't introduce much, if any, extraneous electrical interference to the wire.

For the next few paragraphs, please refer to the following picture, which is a clear, understandable example of a punch down block:

Those grey things in the middle are termination points for single wires. When you deal with punch down blocks, you deal in pairs of wires, and you get one wire to one grey clip. Pretend there is an imaginary line down the middle of that patch panel, because the pairs on the left are separate from the pairs on the right. If you number a line going across, 1 2 3 4, 1 and 2 are a pair, 3 and 4 are an entirely different pair. In the picture, you can tell, because of the numbering scheme they've used. There are 4 clips for a pair, but only 2 clips are wired in the beginning. Each pair is numbered, so the phone company can say "Turn on pair 3044".

Now, at our building, the phone company's lines are on the right side of the wall. On the left side of the wall is another huge array of punchdown blocks. These are for the "house wiring". When they built the building, they pre-ran hundreds of pairs of wires to each floor from this room, so that they wouldn't have to redo it every time someone ordered a T1. Each wire to each floor is terminated to a pair on the left side of the wall in exactly the same manner as the one on the right (including leaving one set empty).

To connect the two sides, you run a twisted pair of wires (it looks like you took a section of cat5, stripped off the sheathing, and just used one set of wires) from the right side of the wall (from the pair of grey clips we left open on #3044) to the left side of the wall (say #514, the 14th pair to the 5th floor, again using the empty grey clips). If you look again at that picture, you can see 3043 has been wired across, because all 4 wires are clipped in, but 3044 has not, since only the rightmost clips have wires.

At this point, you have two wires coming in from the phone company to the punch down blocks on the right. Then you've got wires connecting those punch down blocks to the "house wiring" punchdown blocks on the left. Then you've got vertical wiring up to the floor that the wire ends at.

In the comms closet on that floor (also known as the IDF, or intermediate distribution facility), you have a very similar situation. On the right hand side, you've got the punch down block where the vertical cabling from the MDF terminates, and on the left, you've got a punch down block where the actual wires that end up in your office are terminated. You use another twisted pair of wires to connect the two sides, and at that point, the wire that ends up in your office is connected directly to the phone company, albeit through several punch down blocks and lots of wire.

Now, when it comes into your office, hopefully someone has had the courtesy to install a patch panel for you. The patch panel looks like this on the front:

and this on the back:

As you can tell from the photo on the back, wires are typically matched up color for color when it comes to straight CAT5 cables. When it comes to things like wiring T1s, you're only using two wires, so as long as you remember which one goes to what wire, you're ok.

So, i review, we've got phone company wires coming in, and terminated in the MDF. They're connected across to the house wiring, which is run vertically to the IDF, and from the IDF, it goes to your space. All of this is accomplished with those magic little grey clips.

Now, if only the wires would go in there. It turns out that there's a trick. Or a tool, really, called a punch down tool (creative, eh?). The cheapest punch down tool I've ever seen is a buck. It'll work in a pinch, but the one you want is here:

The way you use it is to arrange the wire you want to punch down against the metal clip. There's a very thin slit in the clip where the wire will end up. Press the tip of the punch down tool against the clip, and push. The spring loaded mechanism (in the expensive tool) or your elbow grease (in $.99 model) will push the wire to the bottom of the slit, and in the process, scrap away the plastic or teflon sheathing on the wire, allowing the metals to make contact. The expensive model will then use the spring action to slice the extra wiring off the end, eliminating extraneous electrical interference (when you're dealing with hundreds of feet worth of cable, this is a good thing). In the cheap model, I'd recommend an Xacto knife to do the job.

As for maintenance, there's not really much that can go wrong in a patch panel, as long as no one comes in and starts pulling on wires. Typically there's a plastic case that goes over the entire block to prevent accidental snags from pulling wires loose.

The best advice is to document everything you can. Leave a hard copy of the documentation in the comms closet so that you can see what's been done. Lots of times, the telephone tech will "tag" the lines that he's installed on the right hand side. The tag usually has the numbers of the pairs that are activated on the telco side, and the phone numbers (or circuit IDs) that match those pairs.

(Photos courtesy of lil 1/2 pint, techmsg, dmitrybarsky)