Update on the Zune issue

Who called it? Who da man?.

Zune freeze issue is a result of the Leap Year.

OK, enough gloating. Hopefully they'll fix the firmware before 2012 ;-)

Note to anyone selling equipment

ALWAYS wipe your equipment before you sell it to anyone.

This includes things like hard drives and network devices.

I can't mention any names at all, or specifics, but I ordered a couple of refurb routers a few days ago, and I was very surprised today when I saw full router configs in place, complete with IPSec settings, ACLs, and plaintext read/write SNMP strings.

Always wipe your configs before you sell the devices. Always.

Zune: Y2K + 8 11.9/12

If your Zune doesn't work this morning, it's not just you.

Apparently, Zunes all over the world froze at or near midnight(local time) last night.

The fact that this follows an iPhone post isn't me gloating, that's just a coincidence. Promise. ;-)

[Edit] It just occurred to me that since this is a leap year, today is day 366. It would be hilarious if they screwed that up.

The mobile device is dead! Long live the mobile device!

My blackberry died at a particularly inopportune moment over the holiday weekend. Specifically sometime between when I went to bed on the 25th and when the firewall cluster members decided to kill each other on the 26th. In any event, I didn't find out about it until nearly noon, which is Not Good(tm).

To rectify the situation, I got permission to go buy another phone, and I cleared it with the president to pick up an iPhone, since there were a lot of positive responses when I asked, way back in November. Matt's response in particular swayed my opinion.

In the intervening week, I have to say that I've become pretty attached to the thing. With some additional apps from the AppStore (if you're considering getting an iphone, prepare to hear that phrase a lot), it becomes much easier to type mail (get Firemail for landscape typing), there are free RDP, VNC, ssh (touchterm), and other apps as necessary, and best of all, there are built-in settings for VPN (IPSec, PPTP, and L2TP). Browsing the web using Safari in landscape mode makes even Opera mini on the Blackberry look like masochism.

In fact, the only complaint that I have is that the notification options suck. Apple really, really dropped the ball with the configuration options for notifications. You can change ringtones for phone calls and text messages, but you cannot change ringtones or adjust volume for incoming email. At all. And the default notification is a quiet, polite "blip", which doesn't wake me up at 3am. And that's a deal breaker.

Before I took my iPhone back, I wanted to try everything, so I decided to jailbreak (I used quickpwn for Windows) it and see what I could change. The process went very, very smoothly (as soon as I realized the the "power" button is the one on the top right that you click to lock the phone).

I used Cydia (the jailbreak equivalent of the App Store) to install OpenSSH on the phone:

Matt-Simmons:/usr root# uname -a
Darwin Matt-Simmons 9.4.1 Darwin Kernel Version 9.4.1: Sat Nov 1 19:09:48 PDT 2008; root:xnu-1228.7.36~2/RELEASE_ARM_S5L8900X iPhone1,2 arm N82AP Darwin

and used that to sftp in and copy the ringtone to my desktop, which I then modified using Audacity to increase the volume and added a double beat to the beginning, so that it now goes "chi-ching!". Saved that, exported it as an AIFF file, renamed it to the original new-mail.caf and then dragged it back across the sftp pipe to the phone. Sent an email to myself, and I'm now guaranteed to be woken up if I get mail at night.

I should really look into getting a dev kit for the phone. It would be really handy to support actual profiles and to use the GUI to set things like this up. There is a local terminal app available, but it doesn't appear to be supported on my firmware. I'm sure it'll be updated shortly.

Anyone else have any neat tricks for a jailbroken iPhone?

Scare Tactics and Security Warnings!

I like looking at big scary apocalyptic events. There's just something...calming...about it. Watching movies where the Earth gets destroyed makes me feel better about the real world and how comparatively un-screwed-up it is. This tendency of mine has spread to the internet, I think. I talked about some crazyness a while back, but today's news is much more fun.

Hackers at the Chaos Computer Conference announced today that they have managed to completely break SSL by using 200 PS3s. Not just that they can spy on communications between hosts communicating over SSL, but that they can brute-force create a "trusted" certificate for whatever they want.

So let me posit a quick scenario. Hackers use the BGP flaw to redirect your bank's traffic to their server, where they've installed a freshly created fake trusted certificate and they man-in-the-middle till the cows come home. Not even two-way authentication can help you then. The best part is that these aren't "bugs" in the applicable protocols as much as flaws in their design.

I suppose in the beginning banks and other lucrative targets can filter known-offenders from their access lists, but the use of botnets will stop that from being an effective tactic.

I wonder if [EDIT] two way PKI will start being cost-effective to implement in that case, since (as I understand it?) the keys and certs aren't being recreated byte-per-byte, they're creating a rogue certificate authority and using that to issue certs. There's a large difference between that and replicating someone's 2048 bit private key. At least, I'm pretty sure. IANAC (I am not a crytpologist).

If the large institutions decide not to do anything, it might get really interesting. Maybe we'll have to go back to writing checks. ;-)

Adventures in VOIP part 2

This is a continuation of Adventures in VoIP part 1


Elastix

The harder half of this endeavor has been the configuration of Elastix. I missed most of the operating system install, but I have been doing a lot of the work getting the extensions set up and configuring the operating panel. My boss got to set up the inbound and outbound routes and configure the trunk lines on the server. Being a Windows guy (and my DOS days being long behind me) I am not all that comfortable working straight from a command line anymore. Thus I attempted to use the web gui supplied with the software.

The web gui is not actually all that bad. I can honestly say they spent some time working on it, but there is one thing they did that drives me absolutely batty. What the hell is up with the red bar? You go in and edit an extension. At the bottom of the form is your standard issue submit button. You think you've made your change, you go and check and nope! It's still the same. You must have missed the red bar. Check out the image. As you can see, the red bar isn't all that red and looks very much like it is a part of the natural background. Up until you look closely and see the pale blue text that says "Apply Configuration" and proceed to facepalm. Unembedded FreePBX (The Elastix form is actually a front end for this) does this right. Notice the orange on blue. Completely contrasting and smacks you right upside the head and tells you that you need to do something. It's noticeable.

Another annoyance encountered dealt with the batch upload. Rather than manually setting up 40+ extensions, you can load a simple csv file and get all of them in at once. After loading (I did remember to click the red button), only some of my extensions worked properly. Oddly enough, only the ones manually entered. I checked to make sure the settings were exactly the same and on a whim, I decided to just hit submit and reload the config. Of course previously unworking extension started to work. I then proceeded to manually reload all the extensions to get the working. I am certain I could have done that from the command line, I just didn't know the way and with my luck I would have just killed something on accident (Yes I have that kind of luck. Ask me about my dead RAID unit sometime, and try not to laugh too hard at me).

With that out of the way, the next task was getting the operator panel online. One thing we noticed is that it could only display 39 extensions in its default configuration. So after a bit of googling, I come across instructions for altering the operator panel. And there is no gui for this. Off to the command line I go. One way a lot of users decide to show more buttons is to physically change how big they are. This option is a no-go for me. Firstly, getting them to look good is a pain in the ass from what I have read. Secondly, our receptionist is somewhere around 70 and her eyes aren't what they used to be (She is surprisingly good with working a computer, as far as receptionists go at any rate. She calls in a timely manner when there is a problem and is polite when something needs fixed.) So to change button positioning, there is a text file to edit: op_buttons.cfg and a perl script to edit: retrieve_op_conf_from_mysql.pl.

The buttons config defines the area that the buttons will take up in pixels on the screen. You can also change column headings, column colors and a few other options. The perl file is where you actually change where the buttons will be. Apparently you edit the perl file so it can generate a buttons config file (op_buttons_additional.cfg) and that file is included with the op_buttons.cfg to get the buttons and their placement. Any manual changes to op_buttons_additional.cfg get nuked whenever Asterisk restarts or the panel reloads. My first attempt at editing the two files was a complete disaster. I found out that it will not automatically extend its default four columns downward, but it will certainly add more to the right off the screen. So that was a dismal failure. I ended up removing the entry for queues (for call queues if you are running a call center) and extending my extensions there.

With that issue solved I moved on to the next one: I was not getting all of my parking lot extensions. For those who have not dealt with larger phone systems (namely me before this job) a parking lot is a set of extensions used for holding calls for other users. That way you can transfer the call there and someone can pick up anywhere in the shop instead of trying to get to their extension before the voicemail get it. Anyway, we have nine parking lots set up 51-59, and the operator panel was only displaying five of them. I double-checked my configuration and I had set up nine, so I delved into the mysterious perl file again and found this:

for (my $i = 1 ; $i <= $numberlots && $i <= 5 ; $i++ )

Now I don't know perl, but I am pretty damn sure I can recognize a for loop when I see it. Two seconds and a reload later and I am in business with all the lots.

And that is pretty much where I stand now. I'll publish further update(s) and anecdotes from the whole process when the system actually comes online.

This was written back on the 18th and since then the system is now online. There will be more forthcoming in this series as soon as I get time to actually write it.

If you're looking for documentation, Craig Borysowich might be your man

I tout documentation quite a lot, but the specifics behind the actual documents have been a little fuzzy. For instance, having an internal wiki is invaluable, but it's as easy to create crap documentation just as well as good documentation (probably easier). What goes into a good document? What form should it take? There are no easy answers.

As with many things, one of the best ways to learn is to examine what other people have done, and that's where Craig Borysowich comes in. If you don't read his blog on IT Toolbox, you should. He consistently produces excellent examples of documentation with his Deliverables series. He also completed posting an example of a system blueprint. For anyone who hasn't already done something like this (like me), it's an amazing time saver, since Craig has done all the hard work.

Like I said, if you're looking for excellent examples of documentation, you owe it to yourself to check out his blog.

Backup (and Recovery)

I'm reading through an enjoyable book called Backup and Recovery by Curtis Preston, and I thought I'd recommend it to any of you who are looking for more information on backup (and more importantly, recovery) schemes. Curtis runs a site called backupcentral.com, which hosts a wiki and forum about backup solutions, commercial and opensource.

I hadn't heard of it, and I figured some of you might not have either.

IT Admin groups on social networking sites

Social networking sites are on the rise, that much is apparent. Tom Anderson sold myspace.com for $580 million dollars. Current estimates are guessing over 140 million users on Facebook. And before you think that social networks are just for kids, Linked In hosts 30 million profiles of experienced professionals who are looking to network with others. Clearly, these sites are tools which can be used to learn and grow in a professional capacity.

I've had my LinkedIn account for a long time, and initially I resisted the others. Eventually I succumbed to Facebook, then myspace, mostly due to peer pressure. Since I have accounts on those three networks, I figured I'd check to see if there were any groups put together by IT administrators. And how.

Most of these groups feature discussions on various topics that you might find interesting. Check them out,and let me know what you think.

LinkedIn
IT Management
System Administrator (Mac, Win, and Linux)
System Administrators
Nagios Administrators

Facebook
Unix Sysadmin
Linux Administrators
Cisco Systems
Appreciate your sysadmin
*NIX
Network/Security/System Administrators
System Administrator Appreciation Day

MySpace
Sysadmin Superstars
Network Admins / Engineers / System Specialists
Computer / Network Administrators
Sysadmins
Network / Sysadmin / Comptechs
Network Engineers

If you know of any other social networking sites (or other types) that you'd recommend, let us know in the comments. I'm always looking for other sources of information, and I know lots of other people are too.

[EDIT]
Talk about coincidence. I wrote this last night and scheduled it for this morning for 8:30am. Before it could go live, Dru posted a link to some BSD Certification groups created on LinkedIn. Funny how things happen sometimes :-)

Systems Administrations Advent Calendar

I really, really thought I linked to this already, but I can't find it, and I'm very sorry! Anyway, it's complete now, so you get to read it all at once. Jordan Sissel put forth superhuman effort this year to start up an Advent Calendar for Systems Administrators. He wanted it to be in the same vein as the Perl Advent Calendar.

Out of the 25 entries, Jordan completed 23 of them. Ben Rockwood wrote Day 17, Time Management, and I wrote Day 23, Change Management. Other than those two, Jordan wrote one a day, each one long and detailed about a different subject, and he did an amazing job. Read through the articles, and I know you'll find them useful and interesting.

Everyone who gets something from Sysadvent owes Jordan a thanks. Please comment on the blog there and let him know you appreciate the work, because he did an amazing job!

Merry Christmas!

First, a little joke I found:

I was musing on similarities between Santa Claus and system administrators. Consider:

• Santa is bearded, corpulent, and dresses funny.
  
• When you ask Santa for something, the odds of receiving what you wanted are infinitesimal.
  
• Santa seldom answers your mail.
  
• When you ask Santa where he gets all the stuff he's got, he says, "Elves make it for me."
  
• Santa doesn't care about your deadlines.
  
• Your parents ascribed supernatural powers to Santa, but did all the work themselves.
  
• Nobody knows who Santa has to answer to for his actions.
  
• Santa laughs entirely too much.
  
• Santa thinks nothing of breaking into your $HOME.
  
• Only a lunatic says bad things about Santa in his presence.
  

Also, here's Admin's Night Before Christmas. Also, the 12 Days of Admin Christmas. There's a Unix Christmas Carol, too.

I hope you have a festive holiday, whichever it is that you celebrate, and stay safe during the ensuing weeks.

"I'm sure it's nothing..just a random nagios timeout error. I'll be right back"

I said that to my wife 3 1/2 hours ago as I climbed out of bed to troubleshoot what I thought was a temporary network latency issue. Hah.

I've spent hours on the phone with Juniper support tonight trying to nail down why my netscreen SSG5 firewalls randomly attack the other cluster members. Tonight was the worst so far. Not only did they fight over cluster master, ns1 actually cratered. It refuses to talk on the network. It won't even see ns2 in the NSRP cluster, even though there's a direct cable connection.

Juniper is sending me the latest firmware, and the colo is going to ship me my firewall so I can re-image it and try again.

Of course, all of that is going to happen at some point after I wake up.

More undersea cables cut...this is not a repeat of a few months ago

Yes, three undersea cables have been severed. It's not like this is unprecedented or anything.

We talked about undersea cables a while back.

Adventures in VOIP Part 1

In the interest of trying to avoid overload, This has been broken up over a couple of days. Read on, and I look forward to your commentary.

Hello again, it's time for another infrequent post by Jim the Windows admin. Since about Thanksgiving, here at our shop we have been getting a crash course in VOIP setup. Our current setup is an old Executone system that has been a work horse here at the shop for at least 8 years before I even got here. Unfortunately, like all old horses, it needs to be put out to pasture. We initially looked at going to VOIP almost a year ago, and to paraphrase the CFO "Limp the current system as long as possible". So of course when November rolls around and the system is restarting itself and dumping all calls 4-5 times a day, we suddenly have funding for a VOIP setup. Go figure. So luckily for us, we have a server we just freed up and our adventures with Asterisk can begin!

The project kicked off with the boss installing CentOS 5.0 and Elastix on a newly spare server. The last 3 weeks have been spent with the 3 man IT department here testing various phones and trying to emulate our current functionality in the new system as closely to the original as possible. For the most part the process hasn't been too bad. We've made a couple of rookie mistakes here and there, but we have a mostly operation system here. Here are a few things I've discovered while working on the system.

Phones

With the rushed time table we were handed, we have looked at 4 different brands of phones for use here in the shop. We looked at the Linksys SPA942, a PolyCom SoundPoint IP 330, a Polycom SoundPoint IP 430, a Cisco 7941 and a little later an Aastra 480i (make a note of this model number, it will come into play later). The dead simplest to set up (and the one the brass here liked the most) was the Linksys SPA942. If you don't need to do a firmware upgrade, you can go from box to working in about 5 minutes, which we in IT thoroughly enjoyed. Of course, after using the phone a bit more, we found that it is not as configurable as some of the other phones on the market (say the Aastra for example) but is perfectly serviceable for what we need. Not to mention it doesn't require text files to do configuration like the Polycom and Aastra phones. As for the Cisco, I'm not sure if we ever got it working. The boss said he would get to it, and I never felt like dealing with it. Sometimes it is good to be the underling.

After doing a little playing around with the setup on the Linksys, we decided to try out its functionality with POE and connecting the phone inline with the between the computer and the wall. To do this, we used a Dell Powerconnect 3448p to provide the POE and we also configured a VLAN for the phones. All was well, up until I decided to change something on one of the phones and my internet connection dropped. Apparently, when the phone reboots (which is does whenever you make a change on its web interface), the power is cut to its ethernet temporarily and you lose the packet forwarding through the phone. Granted it's only momentary, but suddenly dropping your SSH connection to the phone server is very annoying when you are editing a file.

The other thing I found during this process concerns our network map of the shop. The network map of the shop is old. And by old, I mean somewhere in the neighborhood of 6-8 years old. In that time, a new area in the shop was wired up and ports have been added in new places. And apparently no one bothered to label these new ports on the map. Proceed immediately to crawling under desks and trying to read someone's chicken scratch handwriting since they didn't bother with a label maker either. And to top it off, if the port was a low number (ie.6), you had to figure out if it was panel A or panel B. (Apparently starting the numbering where the first panel left off would have been too easy.) All said and done, flashing phones, installing them on desks, and moving ports around took about 3-4 hours and ate most of the Sunday after Thanksgiving day.

Remember that 480i I mentioned earlier? Well, my boss wanted to get a phone for our photo studio that would have multiple handsets since it is a rather large work area. And the 480i seemed to fit the bill. Of course he failed to notice one minor thing. Aastra has a 480i and a 480i CT. The two phones are completely identical in look function (and manual according to the site) excepting of course the ability to pair to a cordless handset. I am still waiting for the right phone to come in.

Work status update

I figure this is a blog, so every once in a while I should just talk about what's going on.

We decided to go with Apptix for our hosted email solution. They seem to have a pretty straightforward interface for adding users and administrating the server. I am a little disappointed that they don't offer an Active Directory import tool, but I'll live. Adding users is relatively painless. Getting all the strange little groups and rules set up will take a bit more time.

My boss is currently in NJ. Since I didn't get the snort machine ready in time for my trip to NJ last week, he's putting it in the rack for me, which I really appreciate. I'll get that into service before too long.

I've been getting behind on tape backups, so I've been concentrating on them this week. I managed to free up a few hundred gigs on an array by finally getting some archives in the tape safe. The freed up space will promptly be used by more archive logs, I'm sure.

It seems like the more vendors I work with, the more sites I have to use that require Internet Explorer. It's a PITA. I've got to load up a VM of some type, log in, load the browser, etc etc, or I've got to remote desktop into another machine and do the same thing. I finally broke down and tried IE4Linux again (which Nick reminded me of). It's improved a lot since I last tried it, and it lets me work on IE only sites. I haven't tried anything too advanced, but it does what I need. The underlying issue is that web client programmers don't understand that a large contingent of technical individuals don't have IE available to them (at least easily). At least I don't think they do.

I'm getting a new tape drive (horray!). Right now I've got an Exabyte (really Tandberg now) 1x10 VXA-3 packet loader. We'll be getting a Quantum Superloader with 16 slots and an LTO-3 drive. It's not as modern as the LTO-4, but it's bigger and faster than our VXA-3 and has 16 slots rather than our 12. The media is actually cheaper, too. This means I'll be needing to re-engineer my backup solution. I'm still deciding between Bacula and Amanda.

My first article has been approved for publication in Simple Talk: Exchange. It'll be in the January issue, which of course I'll link to. Many thanks to my editor Michael Francis for his patience with my horrible writing :-)

And in further good news, I have learned from my boss that they're considering getting more manpower to help me cover the infrastructure here. I don't know what shape or form it will come in, but I'll have a hand in the decision. I'm keeping the name of the blog, regardless of what happens ;-)

Also, Jim is working on a long blog post detailing his recent arguments with his VoIP mess. Look forward to that soon!


So that's been my recent life in a nutshell.

IE Vulnerability - We're stopping use

If you haven't heard, there's a serious IE Flaw which is causing lots of people to recommend temporarily switching from IE to Firefox (or Safari, if you're on a Mac).

Our company just went one further. We're stopping the permitted use of IE. With the exception of those sites which require it, or which we control, we are not permitting our users to browse with IE. Between this and the last few issues (mouse pointer vulnerability anyone?), IE isn't good enough to make us risk the loss of corporate data over it.

Mounting disks by UUID rather than /dev/device

Most everyone who has to knows how to mount a disk in Linux. It's easy, mount /dev/whatever /mnt/whatever. Easy.

What becomes hard for me is when I've got a bunch of external drives hooked up to a machine, and it reboots, how do you make sure everything gets to where it belongs? I used a hack. Every disk has a .disk file that holds the directory name under /mnt/hd where the drive will be mounted. Upon boot up, the server mounts each of the USB disks it finds into a staging directory, checks the contents of the file if it exists, and then remounts it to the intended location. Like I said, a hack.

That is because when I was creating that horrible scheme, I didn't know about Universally Unique Identifiers (UUIDs). Each disk has one, and you can find it by looking at the UUID that lives in /dev/disk/by-uuid/. Each disk has a file in that directory which is a symbolic link back to the /dev device.

It wouldn't be too hard to write a script that looped through that directory, checked if a disk was mounted, and if not, mounted the disk according to either fstab or some other database of known disks.

Anyone using these methods? They strike me as much much more flexible than mounting by device name.

Survey Complete

Well, the IT Admin Job (dis)Satisfaction survey is done, and I'm currently reviewing the results. There were 334 responses to the survey, which is just tremendous. We should definitely get a feel for what other admins go through, and what our work environments are like. To whet your appetite, here are the most popular answers:

1) I deal with:

Server software - 92.8% (309 responses)

2) I am on call

24x7 365 (all the time) - 43.5% (145 responses)

3) I am the primary point of contact in the event of any failure of the IT infrastructure

True - 59.8% (196 responses)

4) The number of users (total) supported in your organization

200+ - 52.1% (174 responses)

5) The number of people providing support to those users

2-4 - 41.0% (137 responses)

6) The number of serverss (physical and virtual) in your organization that are administered

200+ - 21.0% (70 responses)

7) Number of people administering those servers

2-4 - 43.4% (145 responses)

8) Total number of WAN network connections in your organization

2-4 - 30.8% (102 responses)

9) I am...

a) overworked
Agree (118 responses)

b) unappreciated
Agree (107 responses)

c) paid sufficiently
Disagree (129)

d) Happy that I am in my job role
Agree (179 responses)

e) Enjoying my job
Agree (176)

f) Seeking other employment
Disagree (112)

10) Do you think that most people in your position have it better or worse than you do?

Worse - 64.0% (208 responses)


Now, it's important to keep in mind that the raw survey results are just that: raw. There are some untruths that looking at these results might lead you to believe. For instance, you can see that 52% of people have over 200 users in their organization. You also see that 41% of respondents say that there are 2-4 people supporting their user base. This might lead you to believe that it's very common for 2-4 people to support 200+ users, but in reality, when you filter for people who have 200+ users, you see that over 67% of them have 5 or more support people for that user base. 24% of the 200+ people have 10 or more support personnel for their user base.

One thing that struck me was that almost without exception, people thought that others had it worse off than they themselves do. My asking that question was my equivalent to the computer asking Spock how he felt. I wanted to end with a question that might throw people off a little bit, and now I'm glad I did. It's an interesting metric, and people consistently felt despite how overworked and under-appreciated they were, that other people had it worse.

Anyway, I'm working on a much more in-depth report on the results with all sorts of interesting findings (including the one metric that might change an unhappy person into a happy one, and it has nothing to do with money!). Thanks, everyone, for your responses!

Some Campfire Stories

Here are some old stories passed around and down and across. Some are probably tall tales, but they're all interesting in some way, and involve sysadmins trying to recover from problems.

http://www.cavecanen.org/linux/humour/horrorstories.txt

Never thought to check there...

I never thought about it, but there's a Wikipedia entry for Systems Administrators.

It's an interesting view from inside the cage, so to speak.


Also, completely on topic with the subject of this post...

There's sort of like an unwritten rule to networking and systems that states that the strangeness of a problem has a strong correlation to the likelihood that it's DNS based.

We spent a week tracking down an issue with one of our users who could see a file on one server, but not on another, even when both were pulling from the same fileshare.

We made no headway until I tried to access the file. I could log into one of the servers with my newly minted account, but not the other. Using dig, we figured out that one of the machine names was an old DNS entry that should have gotten updated but didn't, so when she logged into the server, newer files than were on the old testing machine didn't show up, but they did on the production machine.

Always suspect DNS for weird issues.

What? SSH stuff AGAIN?!?!?

Apparently the SSH fiasco isn't done. I didn't believe it either, but there are still things that haven't been covered!


Daniel, at Bonetree Blog wrote an overview of a great tool to have in your toolbox: SSH tunnels. Completely aside from the inherent security that an SSH tunnel provides, I've got lots of random hardware (usually cheap routers, APs, and the like) that only want to allow an administrator to log in if the admin is on the same subnet that they are. That's a pain in the butt when you're a couple of states away! To remedy this, I connect to a server that IS on the same network as the device and I create an SSH tunnel through the server to get to the appliance. Daniel explains it better than I'm doing, and he actually uses it to make a SOCKS proxy. Just read his article.

Layers of (non)abstraction

I couldn't sleep the other night, and to try to put myself to sleep, I figured I'd try some remedial programming. I grabbed the Linux 3d Graphics Programming book because it's A) relatively interesting, and B) actually a pretty good primer if you want to refresh the basics of how programming, windowed interfaces, and 3d graphics in general work.

While reading through the section on object factories, a realization hit me. There is an inherent difference in the way that programmers work and systems administrators work. Take the idea of objects, for example. As a programmer, as long as you know the interfaces to an object, you don't have to know anything else at all about the object. You don't need to know its inner structure, you don't care how its variables are defined inside, you just want to know how to access it.

I can't think of a single thing in the whole IT infrastructure that a sysadmin can look at as a black box. Even a server may need to be taken apart and repaired. Heck, I know guys who replace blown capacitors on motherboards. From the base electronics up through logical network design, sysadmins have to cover it all, and the smaller the shop, the more you have to know. I sometimes laugh (and other times lament) the fact that I can be interrupted from enterprise storage design to fix someone's broken mouse, and both are equally valid parts of my job.

I was talking to someone a while back about an open position that they had, and they asked me how wide ranging my experience was. I thought for a second, and said “Do you see that lightswitch? At my company, if it was broke, I'd have to fix it”.

Such is the life of a sysadmin, I suppose. I have wondered if I would even be happy at a place where I didn't have so many different things to do. I think I could somehow manage.


By the way, I'm in the NY/NJ area working this week, so updates may be (even more) sporadic. Just fair warning :-)

Great comment on Bruce Schneier's blog

Bruce put up a blog entry today talking about a one time password generator built into a credit card. It sounds neat, sort of like an RSA security token.

Anyway, in the comments, there was this gem that made me laugh out loud:


Ive never understood how adding 3 more digits to a 16-digit number makes it more secure in the first place. Is this so that if you only managed to steal a copy of the FRONT of a credit card, then you don't reduce the length of the staff of Ra by the right amount and dig in the wrong room? Talk about movie-plot threat!

Posted by: bob at December 4, 2008 7:00 AM


You always get bonus points in my book for referencing Indiana Jones

User Support, Intrusion Detection, and Broken Firewalls. Kids, don't try this at home

I want to start by thanking everyone who has taken the IT Admin Job (dis)Satisfaction Survey. I've gotten around 250 responses in the couple of days it's been up, and that's great. I'm seeing some interesting trends, and I hope to continue to receive responses for the next couple of weeks before publishing results. I'm leaving a link to the survey in the top right hand corner of the blog's homepage (http://standalone-sysadmin.blogspot.com for those of you who use RSS readers), so if you haven't taken it yet, please take a moment and go for it.

I've been very busy at work in the past couple of days, which explains the lack of blog entries. I've been hip-deep in user issues while I've been trying to work on building a Network Intrusion Detection System (NIDS) using Snort. Then I had a firewall cluster member die and try to take the remaining member down with it. It's just been a fun week so far ;-)

If I don't get to write another entry till the end of the week, I apologize, it's just that I usually write my blog entries the night before, and I've been beat and haven't had the energy.

If any of you have killer Snort tips, my ears are open. I'm using the extensive documentation that is available on the Snort site, and also Network Security Hacks, a very fun book to leaf through, and absolutely worth the $20 sale price.

IT Administrator Job (dis)Satisfaction Survey

I drove a lot today, returning home from my holiday weekend. While I was driving, my mind wandered to my job. It was supposed to be my day off, but all I did today was fix things and provide support to users. I was bitter, because I felt like I never have time off, even when I'm off, and that I'm never away from work, even when I'm at home.

Then I started wondering exactly why I was whining so much. It's not like these aspects of the job were unknown to me when I joined the company. At our Christmas party two years ago, my boss told me that part of the reason he hired me was because I told him about the other times I've shrugged off personal life to fix things for work. I guess I should have seen it coming.

I thought back to some other conversations I've had with people, and what their experiences are as admins, and really, the ones I've talked to don't have a clue how the rest of the world works, in terms of numbers of supported devices and supported users, hours worked, etc etc.

To fix this lack, and hopefully to provide some transparency to our profession, I drew up a quick 10 question survey at SurveyMonkey (a great site for building surveys). I call it the IT Administrator Job (dis)Satisfaction Survey. Please take it, it will only use a few seconds of your time, and every result helps to add to the shared knowledge of our positions.

The questions aren't perfect, but I think that they should shed some light. I'll leave the survey open for 2 weeks, till December 16th. After that I'll examine the results, produce some graphs, and post the results for you to see.

Thanks for your help!

RSS aggregation of blog entries and the like

The other day, I read a post on MySysAd blog about people copying blog entries verbatim and passing them off as their own. esofthub added a postfix to his RSS feed to automatically credit the source of the material, which I think is a great solution.

I wasn't too worried about it until I google'd for a random string from one of my blog entries. The results led me to http://www.melonjuice.com/planet, which appears to be an RSS aggregator for technology blogs.

The owner of the site doesn't claim at all to be the author of the material, and I love that my feed is getting aggregated there, but if you look at the site, there aren't any sources listed, except a small link at the bottom with the first name of the author and a link back to the source post.

Like I said, I don't mind, but I have added a 'signature' to my RSS feed which displays the source domain. If it bothers anyone, please drop me a line at standalone.sysadmin@gmail.com and let me know. Otherwise, I'll keep it on. It doesn't seem too intrusive and provides confirmation of the source of the data.

And lest there be any doubt, if you have an aggregator on your site, feel free to aggregate this blog. You're very much welcome to do it, and you don't even have to let me know, though it's neat when I hear from a new site that's doing it :-) Thanks for your support and interest in my material!