Monday, March 30, 2009

When the train comes, everybody rides!

The other day, I mentioned rubber duck debugging, a term with which I had been previously unfamiliar. Since then, I have wanted to pick up a rubber duck to keep on my desk at work, to help me to work through the more difficult issues I encounter (and because it's just funny to have a rubber duck on your desk).

I finally found one the other day. The only problem was that..well, there wasn't "a" rubber duck...apparently they come in flocks or something...

16 Duckies
So yes, I bought the whole kit and caboodle. Walking out of the store, I said to my wife that now I had to find a bunch of people who wanted rubber ducks, since there were 16 in the package, and my issues are 1-duck, maybe 2-duck problems at worst. She thought for a second, and suggested "Maybe your blog readers would like the extras?". Brilliant!

So here's the deal. I bought a pack of 16 ducks:

The whole flock!

The four large ones have been spoken for, so that leaves 12 little ducks. Since I'm guessing more than 12 people will want a duck, I'm going to have to give them out randomly. To help with this, I've created a 4 question survey. 2 real questions, along with your name and email address, so I can contact you if you win a duck. Please know that I will absolutely not give, sell, or relinquish-under-torture your email address. I'll just use it to contact you if my random number generator tells me you that you get a duck.

I'm leaving the survey open all week, since I know not everyone has time to read every day, and I've been way too busy to post normally anyway.

By the way, you don't pay shipping or anything. It's a real, honest to God giveaway. Enter the survey, win a duck, and I'll throw it in a (probably USPS, if you're in the states) envelope and ship it to you.

Note: I make no claims about the troubleshooting prowess of these ducks.

Friday, March 27, 2009

It's only a slow day on the blogosphere

I subscribe to around a hundred blogs. I'm not going to list them all, because a lot of them probably aren't that interesting 75% of the time, and I've linked to most of the ones that continually post good stuff.

It never ceases to amuse me a little whenever I read someone's blog entry that says "You probably noticed that I haven't been posting a lot lately", because unless it's one of a handful of people, I probably didn't miss (or even notice) their lack of posts. I typically wake up to ~40-50 things in Google Reader, and I don't have time to read all of those. I recall offhand that I was sort of worried about Bob when he went on vacation and didn't post anything about it beforehand.

I guess I judge how busy someone is by the number of posts that they put out there, particularly bloggers who have a history of consistently posting good, solid entires. You can sort of tell that Michael has been busy, and I imagine that Jeff has been crushed with work.

Likewise, I have not been posting. AMANDA has been kicking my butt. It took a trip to #AMANDA on IRC for me to really wrap my head around the idea that she really should schedule the full and incremental backups herself, and that I need to let her do it.

That doesn't relieve me of the requirements in the backup policy, though, so I've got to have additional end-of-week backup tapes, too, which means a second scheduled backup set for each discrete item. Anyway, a full AMANDA write up will be forthcoming after I get the solution in place and documented.

And to compound problems, I wiped out my only copy of the configuration I've been working on mid way through and had to recreate it. *groan*

So to summarize, I'm sorry that I haven't been posting as often, though I'm touched if you noticed, and I don't see my current workload slacking off, since my boss is taking vacation next weekend and I'll be pulling double duty on occasion.

Saturday, March 21, 2009

Free Franklin Covey Planner Today

If you've been laid off recently, Franklin Covey is giving away planners to help you get organized in your job search.

These planners are great for Time Management, and have really helped me keep track of my task list.

Just thought I'd throw this out there in case you hadn't heard

Friday, March 20, 2009

Tape Labeling in AMANDA

Alright, I'm once again appealing to people with more experience than I have.

Waist deep in learning AMANDA, I have questions about tape labeling. I'm attempting to find the best practices for labeling tapes, both physically (via barcode stickers on the tape) and the "logical" label written using the 'amlabel' command.

Anyone who knows better, please correct the following overview: For anyone who doesn't know (this included me 2 hours ago), AMANDA uses a program called 'amlabel' to write a label to the beginning of the tape so that it always knows which tape is inserted. It keeps records of every tape that it ever writes, so that in the future when you attempt to recover data, you can insert the specific tape that it asks for.

Since this electronic label can be in nearly any format, many people that I've seen commenting in various places have the knee-jerk reaction to label their tapes "Monday", "Tuesday", etc etc. Many of the old-timers discourage that, because if a tape runs over, you've screwed up the labeling system. Many people suggest "Daily1, Daily2, Daily3", etc etc.

My question is this: What are your tapes labeled, and do your AMANDA labels match the physical labels on your tapes? Also, if you have a barcode reader, do you order customized labels to take advantage of that, so that your barcode matches your AMANDA label, or do you have some sort of mapping between the two in a spreadsheet (or AMANDA itself)?

By the way, if you're looking for customized labels, I found that has some very configurable options. Color, # of characters, etc. Feel free to share your sources if you've got some suggestions!

Thursday, March 19, 2009

Phone Support: What did you say?

If you've ever done any sort of support over a telephone (and I suspect that you probably have), you know that miscommunication can cause big problems, especially when you're having the other person type unfamiliar commands which cause headaches for you when they're entered incorrectly.

I've always wanted to memorize the standard phonetic alphabet that the military uses, because in the middle of a phone call, I hate going "L". So when I was researching it tonight, I found out there there is not just one agreed upon alphabet. In fact, there are several.

Look through the list and find one that you like. One thing I found amusing was that whether the alphabet used names, places, or whatever, almost every one of them used X-ray. I guess there's no getting around that. Except for Norwegian, of course, which gets points for "Xerxes".

Tuesday, March 17, 2009

Excuses for scarce updates

I haven't been updating much in the past week, so I'll do a "blog" entry as an explanation.

Last Friday I flew from Newark to Columbus, and worked all day from the office there in central Ohio. We had a Poweredge 1955 blade's motherboard die before it ever got to be in production, so that will be getting replaced next week.

I didn't go back just to talk to tech support, though. On Saturday, we boxed up all of the equipment from the server room, put it in a U-Haul, along with everything else that wasn't nailed down, and on Sunday my boss and I drove it up to NJ.

Since then, it's been nonstop fun around the office. The HQ office in NJ (the one I work from) used to have a bare four post rack with just a couple of machines in it. Now it houses our backup stack, which I'm soon going to be converting to AMANDA. A blog entry will be forthcoming once it's in place and I've played with it some more. I've found a lot more documentation about backing up with it than recovering from it. If anyone has any killer links to recovery tips, I'd love to see them.

Continuing, I've now got the entire contents of our old office to stuff into the already crowded server room.

I feel like I've worked a ton, even though yesterday was really the only late night. I think it was the not-having-a-weekend bit that did me in. This weekend I'm planning on traveling to the city to check out a museum. I haven't decided between The Met or MOMA. My wife and I did the American Museum of Natural History last time we were in and loved it.

Also, just so there's some sort of sysadmin-y thing in this post, for those of you using Reddit, there is a Sysadmin subreddit. It's actually been in existence for a few months, but had 0 articles (and 4 members) until the beginning of last week. I've submitted 99% of the articles on there as I've come across them, so feel free to join it and start submitting.

How has your week been?

Tuesday, March 10, 2009

Job Hunting? Try this site

There are probably more of these around than I know about (feel free to share your favorite in the comments), but I figured someone here might be able to use an IT specific job site, what with the world economy and all...

[UPDATE] Beware ugly flash as well as crappy spam jobs
I looked at this page with a flash enabled browser, and wow. I no longer recommend going here, but if you've got flashblock and weed through the cruft, there might(?) be ok stuff lurking?

Thanks to DDressler for bringing the problems to my attention.

If you still want to go to the site, here's the url (no longer hotlinked):

I found this on twitter...

Pre-emptive Troubleshooting

Troubleshooting is a very reactive process. By its nature, you're fixing an already existing problem. As good as it is to be able to troubleshoot, it's better to prevent weird problems from cropping up in the first place.

As sysadmins, we have numerous ways to do this. First, as Michael Jenke very sensibly suggests, is to use structured systems management, by administering via script instead of editing files by hand, or worse yet, clicking through the interface.

Another very potent tool that Chris Siebenmann brings up today is using checklists to perform complex tasks. (Incidentally, Chris mentions a term I've never heard before..."rubber duck debugging". I think I'm going to try to expense one of these for troubleshooting purposes)

I'm a firm believer in checklists for anything that isn't (or can't be) automated. On our internal wiki, I have checklists for things like adding and removing users from the infrastructure, adding new machines, etc etc. They're great, and I don't have to "remember" everything that needs done, I can just do it and it's always accurate. And if I've left something off the list, I add it to the list, and it's more accurate.

I wasn't always such a checklist person. It took a while for their usefulness to sink in. Tom Limoncelli does a great job of explaining why in his blog post, Transforming an art into a science, where he explains that back in the bad old days, planes kept crashing until pilots started pre-takeoff checklists. Similarly, has an article about doctors using checklists, which resulted in 36% fewer complications and deaths in the operating room.

Checklists take complex, fun tasks and make them boring.

Monday, March 9, 2009

Short blog outage

Sorry for the time where the front page was unavailable. For some reasons, my Digg buttons went crazy and ate my content. I don't have time to work it out at the moment, but it's back up for now, sans Digg links.

Sorry for any problems!

Another storage option

I have been researching storage for the past few days. I've been concentrating on iSCSI, since I was trying to keep costs down, and a fiber switch is pretty expensive (especially if I want to use it).

While researching, I chanced upon a technology I hadn't heard before: ATA Over Ethernet (AoE). Unlike iSCSI, which transmits the data over TCP, AoE does it via layer 2 frames. This has the implication that, like Fibre Channel, it can't be routed across different networks. For most people, this is not a problem. For some, it's a deal breaker.

In the same way that iSCSI can use software initiators (which turn a computer into an "iscsi server"), there is software available to create AoE "servers". This would be useful if you've got a large machine with many available disk slots.

There are also AoE arrays on the market. Coraid sells some very large arrays. They even offer a ready-made High Availability NAS gateway.

There are drawbacks, of course. There doesn't appear to be a lot (any?) inherent security in the protocol. If anyone reading has experience, I'd be very thankful for some comments as to how host control is done.

I've read comments that were a few years old claiming that it wasn't as stable as iSCSI, but they offered no evidence towards that conclusion, so I have no way of checking to see if their complaints have been resolved.

In the end, I still don't know what I will do, but the more I read, the bigger a blip it is becoming on my radar. What do you think?

Friday, March 6, 2009

The trend towards Overlicensing

Remember when you could buy a piece of hardware, and it was yours, and you could do with it what you wanted?

Maybe it wasn't always like this. I'd like to think that, anyway. I'm pretty young, and have only been in the field for a while, so maybe it was always like this and I didn't know about it.

Here's the deal. I'm in the market for a SAN array. I was looking at Dell's offerings, since that's where we got our AX4-5, and browsing the options for the MD3000i. Here are some options which strike me as...just wrong.

License Keys for snapshots:

  • License key for Snapshot (4 per LUN) AND Virtual Disk copy software features, MD3000i [Included in Price]

  • License key for Snapshot software feature - 8 Snaps per LUN, MD3000 [add $600]

  • License key for Snapshot (8 per LUN) AND Virtual Disk copy software features, MD3000 [add $1,200]

They even make you buy a license key if you want > 16 partitions
  • License key for 32 partitions software feature, MD3000i [add $1,599]

And the storage array isn't the only thing, by far. I have two 16 port fiber switches where I can only use 8 ports. Because I have to buy a license key to use the others. How ridiculous is this?

Does it cost them more money if I want to use more ports? No.
Does it cost less to send out a switch with disabled ports? No.
It's just frustrating as hell.

Of course, it's the same with software licensing. I can buy a license for RHEL, but I can only update the software for the first year, unless I give them another $300.

What is really happening is that installing RHEL is free. An account to connect to the update server is $300 / machine / year.

Similarly, I'm willing to bet that internally to Brocade, the switch hardware is some percentage of the price I payed, and the rest of the price is the license for using the 8 ports I have.

It's still ridiculous, and doesn't endear them to me whatsoever.

Wednesday, March 4, 2009

Systems Administration: Synonymous with Autodidactism?

Autodidactism is a word that you (or at least I) don't hear often. Ironically, do I hear people talk about it a lot, however. To put it simply, an autodidact learns on their own, rather than take coursework or study under a professor. I feel that the simple definition leaves something to be desired.

Learning is, for everyone, a process. A transition from emptiness to rough ideas, and from rough ideas, details emerge. In description, it is a lot like painting. The canvas of our minds are drawn upon, and then filled in to complete a picture of an idea.

Scholarship under a professor, or attending classes on a subject is sort of like paint-by-numbers. The picture is laid out for you, and your goal is to fill in the blanks with the information they present to you. If you are first learning to paint, this isn't undesirable. The structure of the pre-drawn painting guides you and gives you repetition that solidifies good habits. Do it for long enough, and it also constrains you.

Being an autodidact means using a blank canvas, and discovering the picture on your own. You draw the outline, then fill in the open spaces using your own research. How excellent the picture turns out is a direct reflection on your efforts, not the given coursework.

If you are to learn a subject thoroughly, great effort must be made. With the paint-by-numbers course, it's easy to see when you haven't filled in an area, but if you're painting on the fly, it can be much harder to tell that you are missing something.

It seems that no man (or woman) is an island. Until you reach very far in your career as an autodidact, you will be learning from the previous work of someone, whether the the author of a book or an artist on display in a museum. There is no shame in this, though, just as there is no shame in a formal education. Learning is the goal AND the process, so how it is best accomplished depends upon the person.

I am curious what methods you use to research new topics. As for me, I hear of something that interests me, and immediately do internet searches (or write down the subject if I am away from a computer), and go through the various internet sites (invariably wikipedia is included, despite the ill reputation it has among those who only prefer to read about so-called "facts"). I look for e-books, go to the library, and maybe stop by a bookstore or three. Sadly, that is where my progress typically ends. I don't have enough time to research, learn, and acquire every subject that interests me, so I bounce around a lot to various topics. My trip to the Mediterranean was actually the logical continuation of a large segment of my life's learning. I've been interested in Egyptian, Greek, and Roman history since I was in elementary school. Going there and being at the sites, experiencing things I've only seen in pictures before, was absolutely the next level of learning, and it renewed my interest in the subject ten-fold.

I am interested in learning more about how you learn, also. Please, share your methods. If you've never thought about your methods of learning before, take this time to do it. Others will read and gain from your experience, as will you from theirs. Improve yourself by sharing what you know.

I should also state that I started thinking about this post a week or so ago, when I read this blog entry, which absolutely fascinated me. Check it out when you get a chance.

Monday, March 2, 2009

Slashdot discussion on HA and load balancing

As I've said before, I figure that everyone who reads my blog also reads slashdot, but on the off chance you glossed over this Ask Slashdot article, there is a discussion going on of best practices right now regarding High Availability Load Balancing

This is something that I have struggled with myelf. My lengthy fight with, and eventual loss to RH clustering notwithstanding, I've had a lot of success with using dedicated devices to load balance between services. 

I'm interested in hearing the ways you solve this problem. I'll be paying attention to the Slashdot article, because there's almost always a better way, regardless of how you're doing it. 

Dissecting laptops...

Did you know that there are around 35 screws that need removed in order to swap the motherboard out on a Dell Latitude E6500? You didn't? Well, ask me how *I* know...wait, on second thought, don't.

A few years ago, I had to replace a hard drive in an old TI Powerbook. It was ridiculously complex to just take out the drive, involving removal of the entire bottom of the case. Incidentally, if you find yourself needing to take a screwdriver to Apple hardware, iFixIT is a great resource.

Anyway, I thought I would be able to handle the motherboard replacement without too many problems. Boy was I wrong.

I was actually in pretty deep before I decided I had to get some sort of assistance. I had previously Google searched for step-throughs on replacing parts, but I came up empty handed. Midway through gutting the laptop, though, I decided I'd try again. It turned up in the most unlikely of places.

Dell Support

I haven't been able to find exactly that page is linked to from the main site, but it's got to be there somewhere.

That page was directly responsible for me being able to complete the laptop work. There's probably no way I should have even attempted it without more training (and if I knew of the complexity beforehand, I would have recommended hiring a contractor to do it). As it was, it took me 5 hours to complete, but it was a successful swap out.