Today, Slashdot covered a story from MSNBC about unethical high level IT workers. MSNBC reports that 1 in 3 have used "...administrative passwords to access confidential data such as colleagues' salary details, personal e-mails or board-meeting minutes...".
It doesn't specifically say that they were sysadmins, but lets not kid ourselves. We're given a lot of power. As we all learned from Spiderman, with great power comes great responsibility, or at least that's the concept. In reality, people abuse their access. Sometimes it's as innocuous as installing unauthorized software. Other times it's to access corporate financial data.
We're put in a position of trust. We hold the "keys to the kingdom", as they said in that article. It's unfortunate that there are people who would betray that trust, and it's also unfortunate that those of us who wouldn't bear the scrutiny from the people who can't tell the difference.
This is why I think groups like LOPSA(League of Professional Systems Administrators) are valuable. Abiding by their Code of Ethics precludes performing stupid-admin-tricks like spying on corporate email.
If you're interested in joining a professional guild of SysAdmins, both LOPSA and SAGE are valid choices. There's a brief write-up on the history of them by Derek.
Thursday, June 19, 2008
Admin Heroics
You know, 99% of the time, we have a pretty boring job. Sometimes we get to work on interesting problems, or maybe a system goes down, but for the most part, it's pretty mundane.
Sometimes, though, we get called to do relative heroics. Before I was even an admin, I did tech support for an ISP in West Virginia. Once, the mail server went down hard. 20,000 people around the state suddenly had no email, and the two administrators weren't able to be contacted. I was the only guy in the office who knew linux, and it just so happened that I had the root password to that server because I helped the younger admin a few weeks earlier with something.
I reluctantly agree to take a look at the thing, having never touched QMail (ugh), I delved into the problem. Numerous searches later led me to conclude that a patch would (probably?) fix the problem. I explained that to my bosses, and that I had never done anything like this before, but that I thought I might be able to do it without wrecking the server.
They gave me the go-ahead since we still couldn't contact either admin, and the call queue was flooded with people complaining. I printed out the instructions from the patch, downloaded it to the mail server, and applied it as close to the instructions as I could manage. Then, I started the software. It appeared to run, and testing showed that indeed, mail was back up.
I was a hero. At least until the next day when the main admin got back. Then my root access was taken away. Jerk.
Sometimes, we're called upon to extend beyond our zone of comfort. To do things that are beyond our skill levels, and to perform heroics under dire circumstances. These are things that make us better admins. Learning to deal with the kind of pressure that 20,000 people's programs aren't working and it's up to you, or that electricity is down and $18 billion dollars worth of financial reports aren't getting published and only you can fix it. Maybe it's that your biggest (or only) client had a catastrophe and you're the one handed the shovel. Whatever it is, it's alright to think of yourself as a hero.
Because that's what you are.
Sometimes, though, we get called to do relative heroics. Before I was even an admin, I did tech support for an ISP in West Virginia. Once, the mail server went down hard. 20,000 people around the state suddenly had no email, and the two administrators weren't able to be contacted. I was the only guy in the office who knew linux, and it just so happened that I had the root password to that server because I helped the younger admin a few weeks earlier with something.
I reluctantly agree to take a look at the thing, having never touched QMail (ugh), I delved into the problem. Numerous searches later led me to conclude that a patch would (probably?) fix the problem. I explained that to my bosses, and that I had never done anything like this before, but that I thought I might be able to do it without wrecking the server.
They gave me the go-ahead since we still couldn't contact either admin, and the call queue was flooded with people complaining. I printed out the instructions from the patch, downloaded it to the mail server, and applied it as close to the instructions as I could manage. Then, I started the software. It appeared to run, and testing showed that indeed, mail was back up.
I was a hero. At least until the next day when the main admin got back. Then my root access was taken away. Jerk.
Sometimes, we're called upon to extend beyond our zone of comfort. To do things that are beyond our skill levels, and to perform heroics under dire circumstances. These are things that make us better admins. Learning to deal with the kind of pressure that 20,000 people's programs aren't working and it's up to you, or that electricity is down and $18 billion dollars worth of financial reports aren't getting published and only you can fix it. Maybe it's that your biggest (or only) client had a catastrophe and you're the one handed the shovel. Whatever it is, it's alright to think of yourself as a hero.
Because that's what you are.
Wednesday, June 18, 2008
The importance of documentation (and having a place to put it)
As systems administrators, we deal in information. Doing our jobs requires applying skills that we've acquired over the course of our careers, and relating those skills to the systems we come in contact with. In order to do that job correctly, we need to know a great deal of the intricacies of what we're dealing with.
Some of us have small enough networks that keeping everything in mind is tedious but possible. Others have literally hundreds or thousands of servers, and dozens upon dozens of circuits. Most of us are somewhere in-between.
I mentioned it before, in Managing your network contracts, but keeping track of all these details is a task best suited for a computer. There is no sysadmin I've ever seen who wouldn't benefit (or didn't already) from some sort of centralized documentation repository.
Some people choose to write their own. That's certainly a valid choice, if a bit time consuming. At my company, I implemented an internal Wiki, based on MediaWiki. Since wikis are useful for many things, but kludgy for a great many more, I supplement this with an installation of GLPI, which keeps track of all my physical assets, and some logical as well. It's not the simplest software I've used, but it's by far easier than others.
Some of us have small enough networks that keeping everything in mind is tedious but possible. Others have literally hundreds or thousands of servers, and dozens upon dozens of circuits. Most of us are somewhere in-between.
I mentioned it before, in Managing your network contracts, but keeping track of all these details is a task best suited for a computer. There is no sysadmin I've ever seen who wouldn't benefit (or didn't already) from some sort of centralized documentation repository.
Some people choose to write their own. That's certainly a valid choice, if a bit time consuming. At my company, I implemented an internal Wiki, based on MediaWiki. Since wikis are useful for many things, but kludgy for a great many more, I supplement this with an installation of GLPI, which keeps track of all my physical assets, and some logical as well. It's not the simplest software I've used, but it's by far easier than others.
One of the benefits of the wiki is that it allows others in the organization to document what's important to them, as well. Our operations staff has (virtual) reams of paper documenting processes and information. Our corporate contact list is there, as well as the holiday schedule. It's a nice, centralized repository of information for a company that isn't large enough to have a full-fledged intranet web.
The downsides of wikis are that they're primarily edited manually. I have not yet figured how how to reliably import or export information to the database programmatically, other than brute force web-submission. There is also the fact that wikis are (MediaWiki is, anyway, others may not be) anti-hierarchy. The article names are in a flat filespace, so having the same name twice is an issue. It's important to impress this upon everyone creating pages, so that no-one thinks they have page "comments" to themselves. An artificial hierarchy seems to work alright, though, as long as everyone plays along.
In the end, however, it doesn't matter whether you use a wiki or a notebook in the server room. It's important that your assets, procedures, and information base be documented. Never forget that IT infrastructures have bus factors too.
Tuesday, June 17, 2008
The beginning of system management
I was talking to a friend of mine yesterday. He's a junior admin in a Windows shop (not that it makes any difference), but we were discussing the age of his servers, their reliability, and what he was doing about it. I asked the million dollar question: "What would you do if one of the servers died right now?" The answer was chilling, more so to him than me. "I have no idea".
After going more in-depth with the discussion, I learned that, while he did have some general ideas for some services, there was no plan laid out, and what's more, there wasn't even a list of servers anywhere.
We immediately adjourned to Google Docs, where I quickly laid out a spreadsheet with some common fields, and he filled it in. He was surprised. "Wow! We really do have more servers than people".
Maintaining a list of servers is only the beginning, but it's an important foundation for every other part of system and infrastructure management. Until you have your server list, you can't implement host and service checking. You can't really develop a disaster recovery plan until you know what your assets (and liabilities) are.
These are important steps to taking your system to the next level. To really increase availability, you've got to know where you stand.
After going more in-depth with the discussion, I learned that, while he did have some general ideas for some services, there was no plan laid out, and what's more, there wasn't even a list of servers anywhere.
We immediately adjourned to Google Docs, where I quickly laid out a spreadsheet with some common fields, and he filled it in. He was surprised. "Wow! We really do have more servers than people".
Maintaining a list of servers is only the beginning, but it's an important foundation for every other part of system and infrastructure management. Until you have your server list, you can't implement host and service checking. You can't really develop a disaster recovery plan until you know what your assets (and liabilities) are.
These are important steps to taking your system to the next level. To really increase availability, you've got to know where you stand.
Monday, June 16, 2008
Infrastructure upgrades through forest fires
It's funny, sometimes, how we tolerate suboptimal or downright malproductive arrangements in our infrastructures, just because it's inconvenient or inopportune to do it the "right way". It seems like "the right way" either never comes, due to projects getting phased out, or it gets fixed during a cataclysmic upheaval, when it has become an immediate concern.
The case in point is my mail server. We have an A and a B mx record. Originally the B MX just stored mail until the A came back up, then it would get delivered. Everyone checks mail on A, so it can't really be down during the day, and about 6 months ago, the office that B was at relocated and B was never set up. This left us with just A. To make matters worse, A was old enough that it was physically located in our backup site, which used to be our primary site. This was suboptimal. Of course there was talk about moving it to the primary site, but when could a maintenance window be created? And we'd risk the entire period of non-connectivity when it was being moved. No, management said, lets just leave it where it was.
Great strategy. It actually worked fine though, until this weekend.
I came in on Saturday, ready to do some major work on the blade systems I'm building for our new site. I sat down at my desk, ready to dive into work. Since I was alone, Raiders of the Lost Ark was playing on the laptop. I had just logged into the first server when the lights went off, and the telltale screech and whine from the server room told me that we'd lost main power.
In Granville, OH, that's not a strange thing. We've got backup AC and a backup generator, so I wasn't worried. It does have to be manually started, so I jogged into the server room and turned on the CFL floor lamp. At least I tried to. I looked at the generator control panel and it confirmed my fears. No generator power.
I tried for several minutes to start it, but nothing gave me the impression that anything would change, so I called my boss to let him know the situation, and that I was going to start shutting down machines. Since the only critical thing was mail, I suggested that he change DNS to point to an as-yet unassigned IP at the colocation, and that I could setup a postfix process there to queue the mail. He said that it would work, but he suggested an alternative approach.
Why not relocate the physical mail server to the colocation? A lightbulb went off. Of course, not only could I take care of that long standing problem, but because there was no power at all in the datacenter, the normal policy of no-downtime-for-repairs-and-upgrades was out the window.
The next morning, I left work to go home at 5am. The previous 15 hours had been spent completely rehauling the backup datacenter. With the mail relocated to the primary facility, once the power came on in the backup, I had free reign to cull everything unnecessary that had been accumulating.
There is now a pile of cables covering a square yard or so around 6 inches deep of power, ethernet, and copper/fiber cables. There are something like 96 ports worth of switches that I took out, multiple servers, KVMs, fiber switches, and general cruft. The servers are also arranged so that no half-depth servers are hiding between full depth. That was always a pet peeve of mine.
I thought about it while I was doing this, and if fighting normal issues is considered firefighting, then what I went through should have been considered forestfire fighting. And just like a forest fire, good can come from it. It takes the massive heat of a forestfire to crack open some pine cones. It also takes massive infrastructure downtime to make significant changes.
The case in point is my mail server. We have an A and a B mx record. Originally the B MX just stored mail until the A came back up, then it would get delivered. Everyone checks mail on A, so it can't really be down during the day, and about 6 months ago, the office that B was at relocated and B was never set up. This left us with just A. To make matters worse, A was old enough that it was physically located in our backup site, which used to be our primary site. This was suboptimal. Of course there was talk about moving it to the primary site, but when could a maintenance window be created? And we'd risk the entire period of non-connectivity when it was being moved. No, management said, lets just leave it where it was.
Great strategy. It actually worked fine though, until this weekend.
I came in on Saturday, ready to do some major work on the blade systems I'm building for our new site. I sat down at my desk, ready to dive into work. Since I was alone, Raiders of the Lost Ark was playing on the laptop. I had just logged into the first server when the lights went off, and the telltale screech and whine from the server room told me that we'd lost main power.
In Granville, OH, that's not a strange thing. We've got backup AC and a backup generator, so I wasn't worried. It does have to be manually started, so I jogged into the server room and turned on the CFL floor lamp. At least I tried to. I looked at the generator control panel and it confirmed my fears. No generator power.
I tried for several minutes to start it, but nothing gave me the impression that anything would change, so I called my boss to let him know the situation, and that I was going to start shutting down machines. Since the only critical thing was mail, I suggested that he change DNS to point to an as-yet unassigned IP at the colocation, and that I could setup a postfix process there to queue the mail. He said that it would work, but he suggested an alternative approach.
Why not relocate the physical mail server to the colocation? A lightbulb went off. Of course, not only could I take care of that long standing problem, but because there was no power at all in the datacenter, the normal policy of no-downtime-for-repairs-and-upgrades was out the window.
The next morning, I left work to go home at 5am. The previous 15 hours had been spent completely rehauling the backup datacenter. With the mail relocated to the primary facility, once the power came on in the backup, I had free reign to cull everything unnecessary that had been accumulating.
There is now a pile of cables covering a square yard or so around 6 inches deep of power, ethernet, and copper/fiber cables. There are something like 96 ports worth of switches that I took out, multiple servers, KVMs, fiber switches, and general cruft. The servers are also arranged so that no half-depth servers are hiding between full depth. That was always a pet peeve of mine.
I thought about it while I was doing this, and if fighting normal issues is considered firefighting, then what I went through should have been considered forestfire fighting. And just like a forest fire, good can come from it. It takes the massive heat of a forestfire to crack open some pine cones. It also takes massive infrastructure downtime to make significant changes.
Thursday, June 12, 2008
Scripting and trusting GPG
We just added a new client, and like all intelligent companies, we're using GPG to get them their files.
Since we're going to be encrypting them files, we needed to get their public key. There wasn't much issue with that, however several calls to their technical contact have gone unreturned. I'd very much like to sign their key after verifying fingerprints over the phone, but I can't do that if I don't talk to them, and scripting the encryption of a file using an unsigned key is nigh-impossible.
I ended up signing it with a low level of trust, but I'd eventually like to trust it completely.
What are your public key security policies? Would you have signed the key?
Since we're going to be encrypting them files, we needed to get their public key. There wasn't much issue with that, however several calls to their technical contact have gone unreturned. I'd very much like to sign their key after verifying fingerprints over the phone, but I can't do that if I don't talk to them, and scripting the encryption of a file using an unsigned key is nigh-impossible.
I ended up signing it with a low level of trust, but I'd eventually like to trust it completely.
What are your public key security policies? Would you have signed the key?
Friday, June 6, 2008
Managing your network contracts
I've got a fairly complex infrastructure for a small time Admin. At least, I feel like I do.
I've got a point to point between an office and a co-location, I've got 3 office locations, and soon to have two colocations. Soon, one of the offices is going away, and I'm implementing an MPLS network to function as the primary VPN (Juniper Netscreens as the backup).
I've got a total of 5 IP network connections to manage, plus a T1 worth of phone lines at the corporate office, and a couple dozen telephone lines between the other two offices.
Recently, we moved an office from downtown Manhattan to north-central New Jersey, and in the process had to switch carriers. Our telephone provider couldn't give us access in NJ, and our T1 provider couldn't do anything less than a T3 there. Since we're not made of money, we went with DSL there (a big, big mistake. Never trust a best effort service as the primary network connection for any of your offices). I'm almost to the point where we've got an AT&T T1 line in there. I got the IP information today.
Anyway, I've got a lot of connections to keep track of. Currently I have an internal wiki page to keep track of all my connections. I'd prefer something a bit more structured.
Does anyone have any suggestions for managing vendors? Is there software out there that I'm missing, or does everyone just keep it in the electronic equivalent of a filing cabinet (or a real one?).
Comment and let me know!
I've got a point to point between an office and a co-location, I've got 3 office locations, and soon to have two colocations. Soon, one of the offices is going away, and I'm implementing an MPLS network to function as the primary VPN (Juniper Netscreens as the backup).
I've got a total of 5 IP network connections to manage, plus a T1 worth of phone lines at the corporate office, and a couple dozen telephone lines between the other two offices.
Recently, we moved an office from downtown Manhattan to north-central New Jersey, and in the process had to switch carriers. Our telephone provider couldn't give us access in NJ, and our T1 provider couldn't do anything less than a T3 there. Since we're not made of money, we went with DSL there (a big, big mistake. Never trust a best effort service as the primary network connection for any of your offices). I'm almost to the point where we've got an AT&T T1 line in there. I got the IP information today.
Anyway, I've got a lot of connections to keep track of. Currently I have an internal wiki page to keep track of all my connections. I'd prefer something a bit more structured.
Does anyone have any suggestions for managing vendors? Is there software out there that I'm missing, or does everyone just keep it in the electronic equivalent of a filing cabinet (or a real one?).
Comment and let me know!
Thursday, June 5, 2008
Gnotime and Twitter
In an earlier post, I mentioned using Gnome Time Tracker(or gnotime, as it's called) for recording your time spent on particular projects.
I also wrote about the Tower Bridge using Twitter. I started playing with it more, and with the help of this Linux Journal article, I got it working from the command line.
Yesterday I decided to put the two together, so that my twitter status would automatically update whenever I changed different projects.
On gnotime, I have the title as the general task, the description the specific task, and the diary entry as what specifically I was doing. By default, gnotime doesn't allow you to output the diary entry, and I needed that to specify what I was doing on twitter.
Yesterday I downloaded the source, and though I'm not a programmer, managed to coax it into submission. After I got it working on my machine, I submitted a patch to the sourceforge site.
Now when you go to my twitter page, you can see what I was working on and when. Neat!
I also wrote about the Tower Bridge using Twitter. I started playing with it more, and with the help of this Linux Journal article, I got it working from the command line.
Yesterday I decided to put the two together, so that my twitter status would automatically update whenever I changed different projects.
On gnotime, I have the title as the general task, the description the specific task, and the diary entry as what specifically I was doing. By default, gnotime doesn't allow you to output the diary entry, and I needed that to specify what I was doing on twitter.
Yesterday I downloaded the source, and though I'm not a programmer, managed to coax it into submission. After I got it working on my machine, I submitted a patch to the sourceforge site.
Now when you go to my twitter page, you can see what I was working on and when. Neat!
Monday, June 2, 2008
Datacenter tips
Here's a great link that I found by way of Last In, First Out. It's from the Data Center Design blog. The article is Server Cabinet Organization tips.
It's full of great advice. Here's a sample:
4. Use perforated front and rear doors when using the room for air distribution
8. While they are convenient, do not use cable management arms that fold the cables on the back of the server as they impede outlet airflow of the server
Lots more, too. Some may not be pertinent to us, like #12:
12. Have a cabinet numbering convention and floor layout map
If I had a big enough datacenter to need a map, I'm hoping that I'd have been in the job long enough to naturally discover that, but nonetheless, it's a good list.
Here are a couple other links of interest I gleaned from LIFO:
http://www.datacenterknowledge.com/
http://datacenterlinks.blogspot.com/
It's full of great advice. Here's a sample:
4. Use perforated front and rear doors when using the room for air distribution
8. While they are convenient, do not use cable management arms that fold the cables on the back of the server as they impede outlet airflow of the server
Lots more, too. Some may not be pertinent to us, like #12:
12. Have a cabinet numbering convention and floor layout map
If I had a big enough datacenter to need a map, I'm hoping that I'd have been in the job long enough to naturally discover that, but nonetheless, it's a good list.
Here are a couple other links of interest I gleaned from LIFO:
http://www.datacenterknowledge.com/
http://datacenterlinks.blogspot.com/
Google's intelligent searching
Has Google been making anyone else mad with their new "intelligent" search feature?
Example:
I want to determine whether an organization called "Moneymakers Inc" is a scam. I do a search for Moneymakers scam. Google recognizes that moneymakers is two words, so it splits them up and searches for money maker scam. If I want to really search for moneymaker scam I have to type "moneymaker" scam. Irritating.
It's also been auto-conjugating my words for me. The guy who runs the Moneymaker site is named Dick Detering. Google sees Detering, and thinks I'm talking about the verb deter, and it will conjugate it into its various forms and search for those. Again, irritating, and you need quotes to make it stop.
I really don't want a search engine that thinks it's smarter than I am.
[UPDATE]
Since I am now apparently the first result in Google for '"moneymakers" scam', if you've got to this blog entry trying to find out if "Moneymakers Inc" is a scam, I honestly don't know. It sounds too good to be true, but I've not seen anyone complain about them. I also haven't been convinced to give them my money to invest, either.
Drop me a line if you have any info or experience with them
Example:
I want to determine whether an organization called "Moneymakers Inc" is a scam. I do a search for Moneymakers scam. Google recognizes that moneymakers is two words, so it splits them up and searches for money maker scam. If I want to really search for moneymaker scam I have to type "moneymaker" scam. Irritating.
It's also been auto-conjugating my words for me. The guy who runs the Moneymaker site is named Dick Detering. Google sees Detering, and thinks I'm talking about the verb deter, and it will conjugate it into its various forms and search for those. Again, irritating, and you need quotes to make it stop.
I really don't want a search engine that thinks it's smarter than I am.
[UPDATE]
Since I am now apparently the first result in Google for '"moneymakers" scam', if you've got to this blog entry trying to find out if "Moneymakers Inc" is a scam, I honestly don't know. It sounds too good to be true, but I've not seen anyone complain about them. I also haven't been convinced to give them my money to invest, either.
Drop me a line if you have any info or experience with them
Subscribe to:
Posts (Atom)
Posts
