Most everyone who has to knows how to mount a disk in Linux. It's easy, mount /dev/whatever /mnt/whatever. Easy.
What becomes hard for me is when I've got a bunch of external drives hooked up to a machine, and it reboots, how do you make sure everything gets to where it belongs? I used a hack. Every disk has a .disk file that holds the directory name under /mnt/hd where the drive will be mounted. Upon boot up, the server mounts each of the USB disks it finds into a staging directory, checks the contents of the file if it exists, and then remounts it to the intended location. Like I said, a hack.
That is because when I was creating that horrible scheme, I didn't know about Universally Unique Identifiers (UUIDs). Each disk has one, and you can find it by looking at the UUID that lives in /dev/disk/by-uuid/. Each disk has a file in that directory which is a symbolic link back to the /dev device.
It wouldn't be too hard to write a script that looped through that directory, checked if a disk was mounted, and if not, mounted the disk according to either fstab or some other database of known disks.
Anyone using these methods? They strike me as much much more flexible than mounting by device name.
Tuesday, December 16, 2008
Survey Complete
Well, the IT Admin Job (dis)Satisfaction survey is done, and I'm currently reviewing the results. There were 334 responses to the survey, which is just tremendous. We should definitely get a feel for what other admins go through, and what our work environments are like. To whet your appetite, here are the most popular answers:
1) I deal with:
Server software - 92.8% (309 responses)
2) I am on call
24x7 365 (all the time) - 43.5% (145 responses)
3) I am the primary point of contact in the event of any failure of the IT infrastructure
True - 59.8% (196 responses)
4) The number of users (total) supported in your organization
200+ - 52.1% (174 responses)
5) The number of people providing support to those users
2-4 - 41.0% (137 responses)
6) The number of serverss (physical and virtual) in your organization that are administered
200+ - 21.0% (70 responses)
7) Number of people administering those servers
2-4 - 43.4% (145 responses)
8) Total number of WAN network connections in your organization
2-4 - 30.8% (102 responses)
9) I am...
a) overworked
Agree (118 responses)
b) unappreciated
Agree (107 responses)
c) paid sufficiently
Disagree (129)
d) Happy that I am in my job role
Agree (179 responses)
e) Enjoying my job
Agree (176)
f) Seeking other employment
Disagree (112)
10) Do you think that most people in your position have it better or worse than you do?
Worse - 64.0% (208 responses)
Now, it's important to keep in mind that the raw survey results are just that: raw. There are some untruths that looking at these results might lead you to believe. For instance, you can see that 52% of people have over 200 users in their organization. You also see that 41% of respondents say that there are 2-4 people supporting their user base. This might lead you to believe that it's very common for 2-4 people to support 200+ users, but in reality, when you filter for people who have 200+ users, you see that over 67% of them have 5 or more support people for that user base. 24% of the 200+ people have 10 or more support personnel for their user base.
One thing that struck me was that almost without exception, people thought that others had it worse off than they themselves do. My asking that question was my equivalent to the computer asking Spock how he felt. I wanted to end with a question that might throw people off a little bit, and now I'm glad I did. It's an interesting metric, and people consistently felt despite how overworked and under-appreciated they were, that other people had it worse.
Anyway, I'm working on a much more in-depth report on the results with all sorts of interesting findings (including the one metric that might change an unhappy person into a happy one, and it has nothing to do with money!). Thanks, everyone, for your responses!
1) I deal with:
Server software - 92.8% (309 responses)
2) I am on call
24x7 365 (all the time) - 43.5% (145 responses)
3) I am the primary point of contact in the event of any failure of the IT infrastructure
True - 59.8% (196 responses)
4) The number of users (total) supported in your organization
200+ - 52.1% (174 responses)
5) The number of people providing support to those users
2-4 - 41.0% (137 responses)
6) The number of serverss (physical and virtual) in your organization that are administered
200+ - 21.0% (70 responses)
7) Number of people administering those servers
2-4 - 43.4% (145 responses)
8) Total number of WAN network connections in your organization
2-4 - 30.8% (102 responses)
9) I am...
a) overworked
Agree (118 responses)
b) unappreciated
Agree (107 responses)
c) paid sufficiently
Disagree (129)
d) Happy that I am in my job role
Agree (179 responses)
e) Enjoying my job
Agree (176)
f) Seeking other employment
Disagree (112)
10) Do you think that most people in your position have it better or worse than you do?
Worse - 64.0% (208 responses)
Now, it's important to keep in mind that the raw survey results are just that: raw. There are some untruths that looking at these results might lead you to believe. For instance, you can see that 52% of people have over 200 users in their organization. You also see that 41% of respondents say that there are 2-4 people supporting their user base. This might lead you to believe that it's very common for 2-4 people to support 200+ users, but in reality, when you filter for people who have 200+ users, you see that over 67% of them have 5 or more support people for that user base. 24% of the 200+ people have 10 or more support personnel for their user base.
One thing that struck me was that almost without exception, people thought that others had it worse off than they themselves do. My asking that question was my equivalent to the computer asking Spock how he felt. I wanted to end with a question that might throw people off a little bit, and now I'm glad I did. It's an interesting metric, and people consistently felt despite how overworked and under-appreciated they were, that other people had it worse.
Anyway, I'm working on a much more in-depth report on the results with all sorts of interesting findings (including the one metric that might change an unhappy person into a happy one, and it has nothing to do with money!). Thanks, everyone, for your responses!
Thursday, December 11, 2008
Some Campfire Stories
Here are some old stories passed around and down and across. Some are probably tall tales, but they're all interesting in some way, and involve sysadmins trying to recover from problems.
http://www.cavecanen.org/linux/humour/horrorstories.txt
http://www.cavecanen.org/linux/humour/horrorstories.txt
Wednesday, December 10, 2008
Never thought to check there...
I never thought about it, but there's a Wikipedia entry for Systems Administrators.
It's an interesting view from inside the cage, so to speak.
Also, completely on topic with the subject of this post...
There's sort of like an unwritten rule to networking and systems that states that the strangeness of a problem has a strong correlation to the likelihood that it's DNS based.
We spent a week tracking down an issue with one of our users who could see a file on one server, but not on another, even when both were pulling from the same fileshare.
We made no headway until I tried to access the file. I could log into one of the servers with my newly minted account, but not the other. Using dig, we figured out that one of the machine names was an old DNS entry that should have gotten updated but didn't, so when she logged into the server, newer files than were on the old testing machine didn't show up, but they did on the production machine.
Always suspect DNS for weird issues.
It's an interesting view from inside the cage, so to speak.
Also, completely on topic with the subject of this post...
There's sort of like an unwritten rule to networking and systems that states that the strangeness of a problem has a strong correlation to the likelihood that it's DNS based.
We spent a week tracking down an issue with one of our users who could see a file on one server, but not on another, even when both were pulling from the same fileshare.
We made no headway until I tried to access the file. I could log into one of the servers with my newly minted account, but not the other. Using dig, we figured out that one of the machine names was an old DNS entry that should have gotten updated but didn't, so when she logged into the server, newer files than were on the old testing machine didn't show up, but they did on the production machine.
Always suspect DNS for weird issues.
Tuesday, December 9, 2008
What? SSH stuff AGAIN?!?!?
Apparently the SSH fiasco isn't done. I didn't believe it either, but there are still things that haven't been covered!
Daniel, at Bonetree Blog wrote an overview of a great tool to have in your toolbox: SSH tunnels. Completely aside from the inherent security that an SSH tunnel provides, I've got lots of random hardware (usually cheap routers, APs, and the like) that only want to allow an administrator to log in if the admin is on the same subnet that they are. That's a pain in the butt when you're a couple of states away! To remedy this, I connect to a server that IS on the same network as the device and I create an SSH tunnel through the server to get to the appliance. Daniel explains it better than I'm doing, and he actually uses it to make a SOCKS proxy. Just read his article.
Daniel, at Bonetree Blog wrote an overview of a great tool to have in your toolbox: SSH tunnels. Completely aside from the inherent security that an SSH tunnel provides, I've got lots of random hardware (usually cheap routers, APs, and the like) that only want to allow an administrator to log in if the admin is on the same subnet that they are. That's a pain in the butt when you're a couple of states away! To remedy this, I connect to a server that IS on the same network as the device and I create an SSH tunnel through the server to get to the appliance. Daniel explains it better than I'm doing, and he actually uses it to make a SOCKS proxy. Just read his article.
Monday, December 8, 2008
Layers of (non)abstraction
I couldn't sleep the other night, and to try to put myself to sleep, I figured I'd try some remedial programming. I grabbed the Linux 3d Graphics Programming book because it's A) relatively interesting, and B) actually a pretty good primer if you want to refresh the basics of how programming, windowed interfaces, and 3d graphics in general work.
While reading through the section on object factories, a realization hit me. There is an inherent difference in the way that programmers work and systems administrators work. Take the idea of objects, for example. As a programmer, as long as you know the interfaces to an object, you don't have to know anything else at all about the object. You don't need to know its inner structure, you don't care how its variables are defined inside, you just want to know how to access it.
I can't think of a single thing in the whole IT infrastructure that a sysadmin can look at as a black box. Even a server may need to be taken apart and repaired. Heck, I know guys who replace blown capacitors on motherboards. From the base electronics up through logical network design, sysadmins have to cover it all, and the smaller the shop, the more you have to know. I sometimes laugh (and other times lament) the fact that I can be interrupted from enterprise storage design to fix someone's broken mouse, and both are equally valid parts of my job.
I was talking to someone a while back about an open position that they had, and they asked me how wide ranging my experience was. I thought for a second, and said “Do you see that lightswitch? At my company, if it was broke, I'd have to fix it”.
Such is the life of a sysadmin, I suppose. I have wondered if I would even be happy at a place where I didn't have so many different things to do. I think I could somehow manage.
By the way, I'm in the NY/NJ area working this week, so updates may be (even more) sporadic. Just fair warning :-)
While reading through the section on object factories, a realization hit me. There is an inherent difference in the way that programmers work and systems administrators work. Take the idea of objects, for example. As a programmer, as long as you know the interfaces to an object, you don't have to know anything else at all about the object. You don't need to know its inner structure, you don't care how its variables are defined inside, you just want to know how to access it.
I can't think of a single thing in the whole IT infrastructure that a sysadmin can look at as a black box. Even a server may need to be taken apart and repaired. Heck, I know guys who replace blown capacitors on motherboards. From the base electronics up through logical network design, sysadmins have to cover it all, and the smaller the shop, the more you have to know. I sometimes laugh (and other times lament) the fact that I can be interrupted from enterprise storage design to fix someone's broken mouse, and both are equally valid parts of my job.
I was talking to someone a while back about an open position that they had, and they asked me how wide ranging my experience was. I thought for a second, and said “Do you see that lightswitch? At my company, if it was broke, I'd have to fix it”.
Such is the life of a sysadmin, I suppose. I have wondered if I would even be happy at a place where I didn't have so many different things to do. I think I could somehow manage.
By the way, I'm in the NY/NJ area working this week, so updates may be (even more) sporadic. Just fair warning :-)
Thursday, December 4, 2008
Great comment on Bruce Schneier's blog
Bruce put up a blog entry today talking about a one time password generator built into a credit card. It sounds neat, sort of like an RSA security token.
Anyway, in the comments, there was this gem that made me laugh out loud:
Ive never understood how adding 3 more digits to a 16-digit number makes it more secure in the first place. Is this so that if you only managed to steal a copy of the FRONT of a credit card, then you don't reduce the length of the staff of Ra by the right amount and dig in the wrong room? Talk about movie-plot threat!
Posted by: bob at December 4, 2008 7:00 AM
You always get bonus points in my book for referencing Indiana Jones
Anyway, in the comments, there was this gem that made me laugh out loud:
Ive never understood how adding 3 more digits to a 16-digit number makes it more secure in the first place. Is this so that if you only managed to steal a copy of the FRONT of a credit card, then you don't reduce the length of the staff of Ra by the right amount and dig in the wrong room? Talk about movie-plot threat!
Posted by: bob at December 4, 2008 7:00 AM
You always get bonus points in my book for referencing Indiana Jones
User Support, Intrusion Detection, and Broken Firewalls. Kids, don't try this at home
I want to start by thanking everyone who has taken the IT Admin Job (dis)Satisfaction Survey. I've gotten around 250 responses in the couple of days it's been up, and that's great. I'm seeing some interesting trends, and I hope to continue to receive responses for the next couple of weeks before publishing results. I'm leaving a link to the survey in the top right hand corner of the blog's homepage (http://standalone-sysadmin.blogspot.com for those of you who use RSS readers), so if you haven't taken it yet, please take a moment and go for it.
I've been very busy at work in the past couple of days, which explains the lack of blog entries. I've been hip-deep in user issues while I've been trying to work on building a Network Intrusion Detection System (NIDS) using Snort. Then I had a firewall cluster member die and try to take the remaining member down with it. It's just been a fun week so far ;-)
If I don't get to write another entry till the end of the week, I apologize, it's just that I usually write my blog entries the night before, and I've been beat and haven't had the energy.
If any of you have killer Snort tips, my ears are open. I'm using the extensive documentation that is available on the Snort site, and also Network Security Hacks, a very fun book to leaf through, and absolutely worth the $20 sale price.
I've been very busy at work in the past couple of days, which explains the lack of blog entries. I've been hip-deep in user issues while I've been trying to work on building a Network Intrusion Detection System (NIDS) using Snort. Then I had a firewall cluster member die and try to take the remaining member down with it. It's just been a fun week so far ;-)
If I don't get to write another entry till the end of the week, I apologize, it's just that I usually write my blog entries the night before, and I've been beat and haven't had the energy.
If any of you have killer Snort tips, my ears are open. I'm using the extensive documentation that is available on the Snort site, and also Network Security Hacks, a very fun book to leaf through, and absolutely worth the $20 sale price.
Tuesday, December 2, 2008
IT Administrator Job (dis)Satisfaction Survey
I drove a lot today, returning home from my holiday weekend. While I was driving, my mind wandered to my job. It was supposed to be my day off, but all I did today was fix things and provide support to users. I was bitter, because I felt like I never have time off, even when I'm off, and that I'm never away from work, even when I'm at home.
Then I started wondering exactly why I was whining so much. It's not like these aspects of the job were unknown to me when I joined the company. At our Christmas party two years ago, my boss told me that part of the reason he hired me was because I told him about the other times I've shrugged off personal life to fix things for work. I guess I should have seen it coming.
I thought back to some other conversations I've had with people, and what their experiences are as admins, and really, the ones I've talked to don't have a clue how the rest of the world works, in terms of numbers of supported devices and supported users, hours worked, etc etc.
To fix this lack, and hopefully to provide some transparency to our profession, I drew up a quick 10 question survey at SurveyMonkey (a great site for building surveys). I call it the IT Administrator Job (dis)Satisfaction Survey. Please take it, it will only use a few seconds of your time, and every result helps to add to the shared knowledge of our positions.
The questions aren't perfect, but I think that they should shed some light. I'll leave the survey open for 2 weeks, till December 16th. After that I'll examine the results, produce some graphs, and post the results for you to see.
Thanks for your help!
Then I started wondering exactly why I was whining so much. It's not like these aspects of the job were unknown to me when I joined the company. At our Christmas party two years ago, my boss told me that part of the reason he hired me was because I told him about the other times I've shrugged off personal life to fix things for work. I guess I should have seen it coming.
I thought back to some other conversations I've had with people, and what their experiences are as admins, and really, the ones I've talked to don't have a clue how the rest of the world works, in terms of numbers of supported devices and supported users, hours worked, etc etc.
To fix this lack, and hopefully to provide some transparency to our profession, I drew up a quick 10 question survey at SurveyMonkey (a great site for building surveys). I call it the IT Administrator Job (dis)Satisfaction Survey. Please take it, it will only use a few seconds of your time, and every result helps to add to the shared knowledge of our positions.
The questions aren't perfect, but I think that they should shed some light. I'll leave the survey open for 2 weeks, till December 16th. After that I'll examine the results, produce some graphs, and post the results for you to see.
Thanks for your help!
Monday, December 1, 2008
RSS aggregation of blog entries and the like
The other day, I read a post on MySysAd blog about people copying blog entries verbatim and passing them off as their own. esofthub added a postfix to his RSS feed to automatically credit the source of the material, which I think is a great solution.
I wasn't too worried about it until I google'd for a random string from one of my blog entries. The results led me to http://www.melonjuice.com/planet, which appears to be an RSS aggregator for technology blogs.
The owner of the site doesn't claim at all to be the author of the material, and I love that my feed is getting aggregated there, but if you look at the site, there aren't any sources listed, except a small link at the bottom with the first name of the author and a link back to the source post.
Like I said, I don't mind, but I have added a 'signature' to my RSS feed which displays the source domain. If it bothers anyone, please drop me a line at standalone.sysadmin@gmail.com and let me know. Otherwise, I'll keep it on. It doesn't seem too intrusive and provides confirmation of the source of the data.
And lest there be any doubt, if you have an aggregator on your site, feel free to aggregate this blog. You're very much welcome to do it, and you don't even have to let me know, though it's neat when I hear from a new site that's doing it :-) Thanks for your support and interest in my material!
I wasn't too worried about it until I google'd for a random string from one of my blog entries. The results led me to http://www.melonjuice.com/planet, which appears to be an RSS aggregator for technology blogs.
The owner of the site doesn't claim at all to be the author of the material, and I love that my feed is getting aggregated there, but if you look at the site, there aren't any sources listed, except a small link at the bottom with the first name of the author and a link back to the source post.
Like I said, I don't mind, but I have added a 'signature' to my RSS feed which displays the source domain. If it bothers anyone, please drop me a line at standalone.sysadmin@gmail.com and let me know. Otherwise, I'll keep it on. It doesn't seem too intrusive and provides confirmation of the source of the data.
And lest there be any doubt, if you have an aggregator on your site, feel free to aggregate this blog. You're very much welcome to do it, and you don't even have to let me know, though it's neat when I hear from a new site that's doing it :-) Thanks for your support and interest in my material!
Subscribe to:
Posts (Atom)
Posts
