Friday, January 30, 2009

Redundant Redundancy

Many thanks to Ian Carder for this blog entry!

Over the past year we have replaced all of our remaining Cabletron network equipment with Cisco gear. Being a K-12 School District, we have to be extremely frugal where we spend out money. Often, this comes in direct conflict with going the extra mile to make sure whatever system we're deploying has some redundancy in the event of a failure.

Sometimes I can “sneak” redundancy in without having to pay a hefty price or fight for it. This involves simple things like RAID 1/5/10 on servers, dual power supplies, and dual supervisors in the Cisco core switches. We have been making heavy use of ether channel from MDFs to remote closets. That's a cheap solution assuming you have the available ports and don't feel like you have to over pay for Cisco branded GBICs. The other project we're looking to wrap up is creating more that one route to each building in the district using at least a hub network topology. Sorry, no spokes yet! The benefit here is that if we lose a single building, not all traffic is cut off between buildings on either side of the problem building. Lucky for us, we're all on one LAN, so no shared bandwidth to deal with. Just as with the ether channel situation, as long as you have the cabling between sites and available switch ports, it's a cheap proposition. I also have a small cluster running Novell's Cluster Services, but that was something I had to fight for. Beyond that, everything we have is single tier.

So I pose a couple of questions to the faithful readers; what critical services do you need to make redundant and what on the cheap tricks have you come up with to provide the redundancy?

Secular and Religous History in Turkey

Today I pull into Ephesus in Turkey. We actually dock in Izmir, but the majority of the history is in the place that was once known as Ephesus.

This will mark the first visit to one of the 7 Wonders of the Ancient World. The once-beautiful Temple of Artemis has been taken apart piecemeal until it now consists of a single column in a field.

Much more happily, the Library of Celsus has had its facade, along with a portion of its interior. I'm really looking forward to this portion of the trip.

Interestingly, Ephesus is also the home of what is said to be the last earthly home of the Virgin Mary. Amy's family is pretty much exclusively Catholic, so we'll be visiting that and getting some holy water.

Thanks again for reading, and I hope you've had enjoyed this week full of guest bloggers. Next week we'll have more submissions from guest bloggers as well, and I'll continue to update you as to my where-abouts. Take care, and have a good weekend!

Thursday, January 29, 2009

Remote Desktop Shortcuts

Many thanks to Ryan Nedeff for this blog entry!

I love remote desktop, I really do. It's lightweight, it's rather secure, and it just plain works. It's one of the few things that Microsoft really got right. The only problem with it is the issue of keyboard shortcuts.
I'm not a mouse guy. I hate taking my hands off the keyboard. Things like Alt+Tab to switch between programs keeps me from going insane. The only problem is, where does RDP draw the line between commands you are trying to send to the computer you are at, and the computer that you are connected to remotely? Simple, just use a different set of shortcuts.
Here's a list of combinations that I've collected over time. Listed first is the normal shortcut, followed by the RDP equivalent.

Normal Shortcut

RDP Shortcut




Switches between programs from left to right.



Switches between programs from right to left.



Cycles through the programs in the order they were started.



Displays the Start menu.



Displays the Task Manager or Windows Security dialog box


CTRL+ALT+PLUS (+) symbol on the numeric keypad

Places a snapshot of the entire Remote Desktop session window on the clipboard.


CTRL+ALT+MINUS (-) Symbol on the numeric keypad

Places a snapshot of the active window in the remote session on the clipboard.

And, as an added bonus, you can move your RDP session window from Full Screen to Windowed mode with CTRL+ESC.

Καλώς ήρθατε και πάλι!

The subject says "Welcome Back" in Greek (at least, google translate tells me it does).

Today, I'm in Athens, Greece. Last time you heard from me, I was in Rome, which was settled in 750BC, and I talked about how long ago that was. That might be a long time, but it's peanuts compared to Athens.

According to Wikipedia (and we all know wikipedia wouldn't lie to us), Athens has been continuously occupied for 4,500 years. Four and a half _thousand_ years. Think about how long ago Jesus is said to have walked the earth. A long time, right? Double that. Then add 500 more years for good measure. That's a long time. The Roman republic that everyone talks about? Guess where they got the idea? Right. All those fancy Roman pillars and construction methods? They might have shined them up a bit, but guess where they started? Yep, right here in Greece.

So what do you do in a city that's been around long enough that when it was founded there was a different polar star? Eat! Greek food is amazing! I can't wait to get it from the source. And since we'll be eating so much, it's a good idea to walk! Well, it's probably too big to walk the entire time, so we're also taking the bus. There's a route called the 400 line that takes you all over the ancient city for a low daily price. I expect that I'll be getting to know the bus pretty well.

I'm going to have to do a post when I get back and link to all the pictures I'm taking, aren't I?

Wednesday, January 28, 2009

vnstat Console Network Traffic Monitor

Many thanks to Nick Anderson for this blog entry!

Matt was kind enough to ask me to spice things up a bit while he is gone on vacation.(well kind to me, perhaps not so kind to his readers).

If you have ever wanted to monitor network bandwidth I'm sure your familiar with mrtg. I find it useful to be able to do things from the console whenever possible. vnstat fills the void for simple bandwidth monitoring. Since vnstat pulls its information from the /proc filesystem it can be used without root permissions.

To install vnstat on debian just do

sudo aptitude install vnstat

Now you need to create the databases for it to log to. For each interface you want to monitor do the following.

sudo vnstat -u -i iface

After you have created the databases for each interface you want to monitor check to ensure the cron entry was setup correctly. You should have a file in /etc/cron.d called vnstat and its contents should be similar to the following.

# /etc/cron.d/vnstat: crontab entries for the vnstat package

0-55/5 * * * * root if [ -x /usr/bin/vnstat ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then /usr/bin/vnstat -u; fi

Now all you have to do is wait. After some time you can get interesting output like this.

vnstat --days -i wlan0

wlan0 / daily

day rx | tx | total
05.01. 9.69 MB | 1.84 MB | 11.53 MB
06.01. 410.54 MB | 23.72 MB | 434.26 MB %
07.01. 409.89 MB | 25.83 MB | 435.71 MB %
08.01. 68.63 MB | 12.91 MB | 81.54 MB
09.01. 9.75 MB | 3.73 MB | 13.49 MB
10.01. 292.16 MB | 39.78 MB | 331.94 MB %
11.01. 178.23 MB | 1.57 GB | 1.74 GB %:::::
12.01. 350.97 MB | 1.60 GB | 1.94 GB %:::::
13.01. 23.65 MB | 5.89 MB | 29.54 MB
14.01. 410.56 MB | 1.61 GB | 2.01 GB %::::::
15.01. 440.17 MB | 1.62 GB | 2.05 GB %::::::
16.01. 241.75 MB | 26.32 MB | 268.07 MB
17.01. 713.33 MB | 79.22 MB | 792.56 MB %%
18.01. 5.40 GB | 1.64 GB | 7.05 GB %%%%%%%%%%%%%%%%%%%::::::
19.01. 168.10 MB | 1.45 GB | 1.61 GB %::::
20.01. 19.27 MB | 9.64 MB | 28.91 MB
estimated 21 MB | 10 MB | 31 MB

vnstat --months -i wlan0

wlan0 / monthly

month rx | tx | total
Jan '09 9.06 GB | 9.72 GB | 18.78 GB %%%%%%%%%%%:::::::::::
estimated 14.13 GB | 15.16 GB | 29.30 GB

Hopefully its something you can add to your command line toolbox.

Tuesday, January 27, 2009

Don't Panic

Many thanks to Bob Plankers for this blog entry!

What do you do when you first find out that a big problem is happening? Perhaps your data center has lost power, or your intrusion detection system has told you that a web server has been breached by a hacker.

For most people incident handling is one of the most difficult skills to master. You don't get much practice, and most of the time when you do get practice you're actually in the middle of a crisis. You may not have as much information as you need. You might have customers and managers contacting you, yelling at you, wanting to know what's going on, when things will be back up, who is fixing it.

The best advice I've ever been given is simple: don't panic, calm down.

Laurence Gonzales at the National Geographic Adventure Blog describes panic quite well:

"Panic was really useful to us once. We have inherited the structure and function of our nervous system from ancestors who lived in a very different environment, where simple, automatic actions were required for survival. A form of panic -- running away or fighting without thinking, for example -- was apt to keep them alive more often than not."

"But when that quick action also requires logical thinking -- when, for example, you're breathing underwater using a complex apparatus -- it can sometimes incapacitate us. Remember, the higher the emotion or stress, the lower the ability to think in a step-by-step fashion. There are three important steps to take to suppress panic: Breathe, organize, act."

That's excellent advice: breathe, organize, act. Panic causes others to get alarmed, which in turn degrades their ability to think logically with you. You end up wasting time fighting the chaos you've created, rather than calming finding facts and evaluating options. Sometimes you might find, as in my earlier example of an intrusion detection system alerting folks to a breach, that all the panic was for nothing, as it was a false alarm. If you're calm, and the people around you are calm, you can rapidly move on to assessing the facts of the situation, assigning different roles to people, and fixing the problem.

Breathe, organize, act. Great advice for anybody, especially when there's a problem to be solved.

Hello from Rome!

OK, as I'm typing this, I'm not actually in Rome; I'm still at home in my arm chair, but by the time you read this, I'll be in Rome, wandering around the seven hills, trodding on ground which has seen millennia of humanity pass by.

To an American like myself, especially one who hasn't traveled widely, the idea of an area having that much history is unusual. The oldest pub I've ever been in is the Stone Street Tavern in New York City. It's from the mid-17th century, roughly 2,300 years newer than Rome. That's pretty amazing.

I've gotten advice on what to do from everyone that I've talked to about this, from the president of my company to Michael Janke (thanks Mike!), and while everyone has different suggestions, everyone also says that I need to see the Pantheon, so I'll be heading there. I'd like to walk from the Colosseum to the Pantheon, stopping by some local shops on the way.

The best site I've seen for cramming a lot of information in a little bit of space is Their google maps mashup is great.

That's it for now! Next time you hear from me, it'll be Athens, Greece!

Monday, January 26, 2009

My Most Oft Used Cisco IOS Commands

Many thanks to Ian Carder for this blog entry!

Over the past year we have replaced all of our remaining Cabletron network equipment with Cisco gear. Being a K-12 School District, we made use of Cisco Catalyst Express series of switches in our IDFs. The CE series of switches are configured using a web browser. We do have a large install base of Catalyst 2950s and 3550s. Our core network switches, Catalyst 4500 series, are all configured on the command line using IOS. The network configuration is in place and not changing any time soon. Most of the IOS work I do now are simple tasks such as changing switch-port VLAN designations and diagnostics. Here are some of the commands I use most often:

do show run
I always find myself needing to double check something in the running config while I'm in configuration mode. Instead of dumping back into exec mode, issuing do sh run will enable you to check your running config. In general, the do command is to run any exec mode command while in configuration mode.

show run | begin
When you have to check how you have an interface configured on a loaded Catalyst 4500, I find myself hitting space bar a lot and half the time I blow right by the interface or line I was looking for anyway. Assuming I know what I'm looking for and it's a unique entry in the running config, piping the begin command with whatever text you're searching for after a sh run will jump the sh run output right to the text you specified after the begin command. An example I might do would be a sh run | being 6/26 to see the config for blade 6/port 26. You can also combine it with the do command if you're in configuration mode.

show ip route
Generally I don't have to check out the routing table. Our network is fairly stable at this point. Anytime I have to add a new subnet, I'll end up issuing the sh ip route command just to be sure that the subnet has 'propagated' throughout the network. If you're doing more layer 3 work in your network, you'll probably be using the command often.

show vlan
I use this command to quickly see if an access or voice vlan is up, and to see what ports are associated with it. If you create a vlan on a switch, it won't automatically become active unless you assign an interface to it.

show interface
The show interface command is extremely useful for diagnostics. There is a ton of useful information for each interface that you run this command against. The most basic output is whether the interface is up or down, and whether you shut it down(administratively down) or not.

If you find yourself schlepping around IOS often, or any other platform for that matter, feel free to talk about commands you use most often and why.

Can you believe, I got over the wall!

Hello, and welcome to a very unusual fortnight on this blog.

Typically, I talk about server and network administration, I harp on documentation, diagramming, user support, and a wide variety of other admin-y topics. Not for the next two weeks, however.

You see, for the next two weeks, I'm on vacation.

My wife and I are in the middle (near the end, actually) of relocating from central Ohio to New Jersey, and we desperately needed a vacation. It got to the point where I didn't care at all, and would have gladly spent 2 weeks vegging in front of the TV rather than go to work or respond to emergencies. Luckily, I got creative and starting trying to come up with places to go or things to do during the two weeks. After tossing around and discarding several ideas, a European vacation sort of got stuck in my mind.

After weighing the financial options, we decided on a Mediterranean cruise. It had all of the features we wanted in a vacation: travel to several interesting places, no (or extremely expensive) cellphone reception, and lack of proximity to work or a computer.

Since we hadn't had a vacation at all since our honeymoon in 2007, we went for a 12 day trip all around the Mediterranean. Our ports of call are Rome, Athens, Ephesus, Alexandria (which we're really just using as a gateway to Cairo), and Malta before returning to Barcelona.

If you're into cruising and are curious, we're going to be on the Norwegian Jade.

Now, since I have absolutely no intention of using a computer more complex than the one in my camera, I'm not going to be writing blog entries for the next two weeks. Rather than leave the blog barren, I have enlisted the aid of some excellent bloggers, as well as some readers who have shown interest in contributing material. This will serve the dual purpose of providing interesting material and informative content, and a link back to their blog, so if you aren't familiar with their work, you can check it out and see what they're all about. If you already read lots of blogs, maybe you'll find your favorite author doing a guest piece here.

So for now, enjoy the guest bloggers, and have a good two weeks on the internet without me.


--Matt Simmons

Wednesday, January 21, 2009

This means I've been here too long...

"...domain controllers are sort of like hermit crabs. It sounds like there should only be one, but in reality they like to run in packs"

Good knowledge resource for Windows AD stuff

I've been using the Petri knowledge base a lot, and I thought someone else might find it useful. I particularly like their Active Directory section. I don't know a lot about it, so it's useful. I need to go back and reread some AD books.

Sorry also for the dearth in content lately. This is my final week in the Ohio office, so I'm struggling to get everything done that needs to be completed before I move (and bring some equipment with me). I'm trying to get everything here at work caught up, as well as everything at home packed up and ready, getting my new apartment done in NJ, and making sure that things are going smoothly for my two week vacation (starting this coming weekend: expect an update Monday morning with more details). My wife Amy has been unbelievably helpful in taking care of things at home, which lets me concentrate on stuff at work, but there's always bleedover between the projects. Things will be better soon. :-)

Saturday, January 17, 2009

Slashdot: Active Directory Alternatives

If you're in the market for a directory server, there's some good discussion going on over at Slashdot regarding Active Directory and alternatives to it. I know not everyone has the resources to build a Microsoft-based infrastructure, so if you're still wanting the centralized administration, you might find something useful in that thread.

This is the sort of thing that happens when you google yourself

Have you ever read a story or piece of news, and the person in the story had your name? You know that kind of out-of-body experience you get for a second? Yea, I just had that happen.

The weirdest thing was that the occupation in the article was even a sysadmin. The only thing weirder than this has to be finding Rule 34 of yourself. *shudder*

Thursday, January 15, 2009

Enterprise Quality Firewall Products?

So, I'm shopping around for an enterprise quality firewall product to replace my Juniper SSG5s which have caused me nothing but grief.

The obvious choice (to me, anyway) would be the Cisco ASA line. The current model is the 5500 Series, which seem pretty full featured. The 5510 with Security Plus seems to fit my needs almost exactly, if you add on the Advanced Inspection and Prevention Security Services Module (AIP SSM). Of course, prices aren't listed.

I'm curious about other options. I see that Juniper has their Netscreen 5200/5400 series, but I'm really pretty tired of dealing with Junipers. My latest issues have probably caused that jaded outlook, and these 5x00's probably have nothing at all in common with my little SSG5s. If you know, please drop me a line.

My question is, are there other vendors that I should be looking at? What am I missing here to make an informed decision?

Tuesday, January 13, 2009

What do you need to be a sysadmin?

There is an interesting thread happening on Reddit right now. Eduran asks "What do I need for a career in sys admin"?

He continues:

"What kind of schooling do i need if i want to work in linux administration? I already have a couple years in linux experience (messing with Arch and basically just soaking things up like a sponge), and the linux world fascinates me to no end."

What advice would you give him? Definitely comment in the thread if you've got a reddit accout. I'm adding a link to this post in the conversation. Chime in! How did you get started?

Sunday, January 11, 2009

Switches in Server Racks

Finally someone else ever-so-briefly enters into the physical side of IT administration.

Rick Vanover discusses whether or not to put switches in server racks. In case that sounds like a really really dumb question, he's talking about very large, datacenter sized installs.

It's nice to see some sort of discussion about the physical arrangements of parts instead of just assuming that everyone knows the right way to do it (or even that there is a right way to do it) or that it doesn't matter.

Friday, January 9, 2009

Learn to not suck at security

DaveB on ITToolbox has a post today entitled How to suck at security. It's a short list, but it links to a longer one.

I really like the list format "How to suck at *whatever*"

Sometimes you don't ever look for the right way to do something until you learn that you're doing it the wrong way.

Thursday, January 8, 2009

Adventures in VOIP part 3

For those who may have missed the first parts, you can read them here and here.

So on Dec. 18 we rolled out our VOIP setup. Not the greatest time from a learning curve standpoint (who doesn't forget things over a long weekend), but it made perfect sense from a business standpoint. It was just slow enough to provide stress free time to learn it. And it worked out that I had no other projects so I could sit with the receptionist and make sure she had a handle on what was going on. In these first few weeks have been interesting since we are finding out all sort of things that were overlooked and just plain overlooked during our rushed planning/implementation cycle. I think a few people are starting to learn why we asked to do this last year...

Anyway, the first complaint registered came from the receptionists (our payroll persons doubles as a receptionist when things get busy). When you rush through things, you tend to implement things before you fully understand the ramifications for doing so. As mentioned in the earlier adventures, we went with Linksys SPA942 phones. While being dead easy to setup, we did realize that they were less flexible than some of the other alternatives that we looked at. The problem the receptionists noticed deals with the placement of transfer and blind transfer. For some pointless reason, if you have four buttons on a phone for options, Linksys decided that you need to have two options immediately available and two more available from another one. Unfortunately one of these other options is blind transfer. For those who do not answer/route calls on a daily basis this means you have additional button presses just to dump off a call (not so bad). It also means that the caller ID is not passed on either (bad). So every call transferred through looks like it is coming from the receptionist and not from the outside world. From what my boss tells me, complaints have been registered with Linksys by more than a few people about this. We're hoping they fix it in the next firmware release, but if anyone has a good solution, please let me know.

The second complaint from the receptionists is that the Flash Operator Panel (FOP) does not show Do not Disturb(DND) or Call Forward(CFWD) status. Our old system could handle this (it underlined the extension if someone turned on DND/CFWD), but for whatever reason this feature is not implemented natively. I stumbled across a solution a few weeks ago, and it is currently on my list of things to do.

Our third complaint was actually registered by much of the shop. On our old system, dialing an extension used the intercom function on the phones. So if I dialed a department I pretty much just started talking. Now, the phone just rings. When a call rings more than 20 seconds at the receptionist's desk it will ring through the shop. So unless you are paying attention to your caller ID, you could end up talking to a customer that you really did not want to talk to. We haven't decided how we are going to fix it yet. We're leaning towards either configuring another line for internal calls and putting a different ring tone on it, or going with intercom functions to emulate the old functionality. This is also on my todo list, but somewhere near the bottom of it.

Complaint number four brings us back to one of the receptionists. She was telling us that there were not enough lines available on the phone. Well, thankfully I have a solution to implement this time. The Aastra 480i phones I mentioned in the first go round will no longer gather dust. With their ability to handle nine lines, I am just going to install those for them once I get them completely configured/figured out.

The rest of the issues I remember off the top of my head all come from us here in IT. The distro we are using, Elastix, just does not seem to be fitting out needs properly. When we started the project we knew there were other alternatives, so we broke down and installed one of them. In this case, we installed Trixbox. It has a lot in common with Elastix, with the added bonus of better support forums and actual support from several major phone manufacturers. Thankfully, it is also similar enough that I can just direct port some of our configurations. So testing is commencing on that for now with a good chance that we will be porting over to it some time in the near future.

This has been a real learning experience for us here, and I am glad to be working on it (even if I am sick of phones already). So if any of you have any more tidbits or pointers, please feel free to share.

Tuesday, January 6, 2009

My first article is up at Simple Talk: Exchange

My first column is up at Simple Talk: Exchange. If you like it, make sure to vote it up! I hope you get something from it. You might also look at my sysadvent contribution as a peripheral to this.

Thanks very much to my editor, Michael Francis. He worked with me to get a format that we think works very well. I'm looking forward to being a regular contributor to the ST:E. Thanks also to all of you, my blog readers. You're the reason I started writing the blog, and what has kept it going over the first 9 months.

You can sign up to receive the monthly SysAdmin / Exchange newsletter here (that will contain my article and others) and get the free ebook Sybex’s Best of Exchange Server 2007 today.

Use Double Driver to simplify your next reinstall

I honestly don't remember what I was looking for today when I came across Double Driver, but I don't think it was anything as cool as what I found.

Double driver is for Windows systems, and when you run it, it scans the system to find all the used drivers. It then gives you the ability to backup those drivers (maybe to a network drive or flash media, for instance).

I installed it and checked it out, and I now have a flash drive that contains all the drivers that I'd otherwise have to hunt for and find if I need to reinstall my Vista machine. It's a very handy application, and if you know you're going to be reinstalling (or even if you don't, it doesn't hurt to be safe), you might want to check it out.

See, this is why I like open source software

I'm working on learning more about Microsoft's Sysinternals. There are a lot of really handy sounding utilities, and too many for me to keep track of. I was looking through the disk utils, and I found LDMDump. It's a utility that prints out information on the logical disk scheme. Here's a quote from the utility's page:

"There are no published APIs available for obtiaining detailed information about a disk's LDM partitioning, and the LDM database format is completely undocumented. LDMDump was developed based on study of LDM database contents on a variety of different systems and under changing conditions."

Now, I ask ridiculous is it that a Microsoft-paid developer has to resort to essentially reverse engineering a partitioning scheme to figure out how it works? "The LDM database format is completely undocumented". Unreal.

Monday, January 5, 2009

Debugging traffic flow in netscreens

I use Juniper Netscreen (5GT and SSG5) to maintain the intersite VPNs in my company, and to function as firewalls. I was having an FTP issue today, which I suspect is caused by the firewall, so I wanted to see what was going on.

If you work on several pieces of equipment that are similar in function and interface, you might get confused, or your brain takes a shortcut, and you end up doing something silly, like typing "ls" at a Windows command line. Same thing with me and routers sometimes, so I logged into my netscreen and typed "debug" and hit question mark, because I wasn't sure of the argument list. The list of arguments came up, and I started scrolling through , looking for likely candidates. Around this time, it hit me, "There is no debug statement in ScreenOS". I quit out, and just hit "?", which should give me a list of all available commands. Sure enough, it wasn't listed there:

alpha:ns1(M)-> ?
clear clear dynamic system info
delete delete persistent info in flash
exec exec system commands
exit exit command console
get get system information
ping ping other host
reset reset system
save save command
set configure system parameters
trace-route trace route
unset unconfigure system parameters

Well...huh. So I googled it. It turns out that there is a debug command, just not generally documented, and it can do what I'm looking for. I found a blog entry on Geek2Live that seemed to hold the general ideas of what I wanted, and it even included a nifty mindmap to explain it.

If you're interested in this sort of thing, you might enjoy this list of hidden ScreenOS commands and the Juniper knowledge base article on capturing debug flow information.

Sunday, January 4, 2009

Catagorized screw heads

Most people have to deal with two types of screws: flat heads and cross-heads. Sysadmins are not most people.

I've dealt with hex heads, security bits, weird triangle things, and stuff I could hardly describe. Someone, however, has gone to the effort to categorize different screw heads. Here's the writeup on instructables.

Anyone want to start bragging about how big their security bit set is? ;-)

Thursday, January 1, 2009

Cisco Feature Navigator for IOS images

Ever look at the filename of the Cisco IOS image on a router and wonder what all the letters mean and what capabilities are included in your OS?

To find out, you can go through a few dozen pages in the IOS Reference Guide, or the much longer Cisco IOS Packaging (Product Bulletin No. 2160).

If you want a quicker way to find out what's in your image, you can use the Cisco Feature Navigator. It's really simple. You put in the image name that your device boots from and it gives you a list of what features you've got. Figuring out those individual features is left as an exercise for the reader ;-)