Security Policy Best Practices

We're taking on a new client, and their standards requirements are a bit beyond anything we've ever encountered.

I just finished going through a nearly-600 line spreadsheet answering questions about our network's physical and logical security controls. It also included questions about our in-place security policies and what was covered by them. Nothing in the questionnaire was a bad idea to have implemented, but the sum total was a bit overwhelming.

We do have a security policy; it's covered in a page or so of the employee handbook. It's implemented, as well, but beyond that, everything has been done according to a relatively "common sense" approach. To remedy this lack of standards, my boss and I are currently going through the SANS Institute's "Security Policy Project", and I've got to say, it's more overwhelming than the original spreadsheet.

We've accumulated about 15 documents that we think apply to our situation, and we're in the middle of revising them and customizing some of the technologies they cover to work for us. After they get drafted and approved, I get to implement them. I can't wait.

The hardest part will be a change in mindset for the users. I can't wait to see how the operations side responds to this. I suppose it's the price you have to play for running with the big dogs, so to speak.

Which brings me to my question. Does your company have a codified security policy? Do you ever do spot checks, or audits? Do you abide by the policy?

As always, I have anonymous comments enabled, so feel free to comment as such if you are worried about revealing too much about your network.

Life After Windows?

This isn't particularly related to systems administration, per say, but I think that the people who read this blog might be interested in it.

According to SDTimes, Microsoft is preparing for life after Windows, with an operating system called Midori.

When I first started reading the article, I was intrigued. Windows has been the prevalent operating system of home computers now for almost 15 years. Before Windows, there was DOS. That was a paradigm shift. I'm wondering how extreme the next jump will be. How far will Microsoft push the line?

I was sad when I got to this sentence:
One of Microsoft’s goals is to provide options for Midori applications to co-exist with and interoperate with existing Windows applications, as well as to provide a migration path.

Unfortunately, that requirement might stop them from doing anything terribly exciting. With a decade of legacy code to support, I don't see how the change could be earth shattering.

I'm not a Microsoft hater. I don't use Windows, and I don't particularly like it much, but I respect what it has done for standardizing the personal computer industry, and I'm interested to see if Microsoft can push the bar, and really come up with something good. Time will tell.

Arranging your workspace

When you sit down at your desk, how is your comfort level? Do you have to shove piles of paper out of the way, or do you believe that a cluttered desk indicates a cluttered mind? Do you have enough room to work? Are your ergonomics in place to stop you from getting headaches, wrist injuries, and the like?

Ryan and I were talking the other day about how our desks were arranged. He was trying to fit a laptop on the same desk as his dual monitor setup, and it struck him that it was nearly impossible to search for how people arranged their desktops without getting screenshots of, well, desktops.

In the interest of sharing information, I present a rough diagram (created in OmniGraffle), for your enjoyment:



In the interest of professionalism, I have left off the trilobyte fossil, Michael Scott sticky-notes, and bobble head Knight Who Says Ni. I also forgot the aspirin, pepto bismol, tums, and multivitamins. Being a sysadmin is hard sometimes :-)


Depending on the number of projects I'm working on, my desk is pretty messy, but filing generally helps reduce clutter.

How are your desks arranged? Do you have a way of organizing that you like more than others? Throw pictures up on flickr and share!

Fascinating behavior from a linux host

Is this undocumented behavior, or am I missing something obvious?

I am having a very strange issue with one of my Linux hosts.

The primary symptom is that the host responds on two IPs, when ifconfig clearly shows that eth0 is the only interface up (besides the loopback), and only has one IP address.

The arp table on each host that connects to the machine in question shows the same MAC address, regardless of which IP you connect to.

Pinging either IP address from the machine results in responses of such a small amount of time (0.034 ms) that I have no doubt that packets are not leaving the interface.

Running ipconfig -a indicates that eth1 is assigned the alternate IP address that the machine is responding to, however all indications are that the line is down and upon physical examination of the machine, I have verified that there is only one ethernet cable plugged in, and it is going into eth0

At this point, I'm not really sure how to proceed. Obviously I can remove the alternate IP address from eth1, but I'm still at a loss as to why the machine
A) responds positively to an IP address assigned to a down interface
B) sends that request through another, up interface
C) responds with the MAC address of the up interface

I've checked, and there are no iptables rules, there are no specific routing instructions, and neither interface shows up in the arp table of the host in question. All of these are proper conditions for this host.


Any comments or suggestions?

Proper Disclosure of Technology

As you might have read, I recently moved some equipment into the NJ colocation. One of the documents I generated for that move was the rack diagram, showing all of the equipment being installed, and where in the rack it went.

Being sort of proud of it, I showed it to some people in the corporate HQ when I was in the office there, and the CEO saw it. He asked, "This is all the equipment in the rack?". I verified that it was, and he said, "Good. Now get with X (the head salesman) and work on text describing this for him. I want to use it in marketing materials."

Now, on a certain level, I don't mind selling the company's product based on our technology. In fact, I'm pretty proud of what I've managed to put together, and I think if you're going to throw almost $100,000 into technology, that technology should help you actively recoup the expenditures.

My main concern is security. I'm certainly not someone who relies on security through obscurity (although it never hurts to have some of that, too), but I'm questioning what information I should release.

I've gone to measures on this blog to not reveal the name of the company that I work for, mostly because I don't think it's important to the blog itself, but also because I'd rather not reveal the internal structure of my company to anyone interested in learning more about it. It's none of their business.

In that same light, I don't really want sales material handed out stating that I've got 2 Juniper SSG5s setup in a cluster configuration, and that when they hit our website, they're actually talking to high availability Kemp LoadMaster 1500s. If I've done my job right, even with that knowledge, they wouldn't be able to break in, but it's still more information than I feel is comfortable.

The path that I'm leaning to not having the sales guy release any of the diagrams I've made this far, and not mention any of the specific technologies we're using, but only vague generalities. "High Availability clustered firewalls" instead of SSG5s, and "multiple redundant load balancers" rather than LoadMaster1500s. I haven't decided what I want to do about operating systems. Personally, I think the fact that we're a linux house means that our servers are more reliable. I'm sure a Windows admin would feel the opposite. I suppose it's much like any other divisive choice, and that polite conversation should steer away from it. Religion, politics, income, and operating system choice.

Any ideas on how you or your company approach this issue (or how you would, given the chance?)

Happy Sysadmin Day

As The Life of a Sysadmin pointed out, today is Sysadmin Day!.

A big "thank you" to all the administrators who helped my packets reach blogger.com, and let me post to my heart's content.

HOWTO: Server Cable Management

Introduction
Happy Friday to you, and welcome to the (semi, almost) weekly how-to. Today, by popular request, we're talking about cable management.

Good cable management defined is making cables, cords, and wires cleanly and neatly arranged, and it facilitates easy tracing of cables and simplifies replacing bad cables. A lot of time, these are all mutually exclusive, so we concentrate on the task at hand and taylor our cable management to that end. It also promotes good air flow in the back of the machines, which should be reason enough to do it.

Cable management is, to a lot of people, something you would rather be done with than do. It's also far easier to do when you don't have to take down the network or power from a running system. For this reason, it's important to keep it in mind in the beginning, when you're figuring out how you are going to cram all those wires wherever they need to go. The job isn't made easier by the huge number of places wires have to go, nor by the types of wires that you're dealing with. Lets break it down a little and see what the best policies are for each.

Shelved Computers

Whether this is a scene from your work, or your basement, almost all of us have been there. Industrial shelving with tower servers everywhere. Wild tangles of cable in the back, probably inaccessible against the wall, where they attract dust bunnies and cobwebs. It's not ideal, but it's all a lot of us have to deal with. What do you do to make it useful?

First, if your shelves are against the wall, pull it out at least a foot, if at all possible. The amount of heat being built up in the back is immense, and with nowhere to go, you're just baking the back of your equipment. Once you've got some space back there, you can go to work.

The major problem with industrial shelving is that it's not meant to hold servers, and there-for it has no cable management. It's easy to add your own. I have several runs of plastic cable runs that are open sided, with fingers to hold cable. Here's an example:


They can be several feet long, and work great on industrial shelves. Just mount them to the back of each shelf layer, and run your cables through there. Since they have the fingers, you just pull the cable out of the run and into the server where-ever you need. It works great, and looks neat and tidy.

It is important to remember that long runs of power cable should not be mixed with network cable. It would be best to run your ethernet cable along the top shelf, and power across the middle shelf. That would keep them separated, so as not to create massive packet loss.

If your cable duct is like those in the picture, with a "lid" to keep the cables contained, then great! Use it! They snap on and off easily, and really improve the look and airflow. They do make cheap ones without that cover though. I know because I have several. In that case, you're going to want to secure the cables together, so they don't fall out of the channel. For this task, lots of admins use plastic zip ties. That was my preferred tool, as well, until I discovered that velcro ties. They're just as easy, reusable, and you don't have to carry wire cutters to snip that last 6 inches of extra plastic. These qualities pale in comparison to the fact that, when you need to replace a cable at 3 in the morning, you don't need to find the wirecutters that you carry with you during the day. It's just "rriiiipp" and you have access to your cables. I cannot recommend these highly enough for any cable runs that you foresee replacing.

Typical Server Rack

The other major form factor for servers is rackmount. If you've never used it before, you might want to check the introduction I wrote a while back.

Since racks ARE meant to hold servers, all but the least expensive have built-in cable management. Typically there are channels running vertically up and down the back corners with access points placed intermittently along the way. Ideally, all your servers would have power cords on one side and network cables on the other. This is pretty much never the case. Still, I have all of my ethernet on the right side (if you're standing looking at the rack from behind it), and all my power on the left.

One note, when it comes to installing servers in the rack. Lots of servers (Dell, I know for sure. HP maybe) come with cable arms for the back of the rack. They flex and pivot, and are meant to allow you to pull the server out the front of the rack without having to unplug any of the cables. If your rack is low density, they're only mostly irritating, but if you've got several machines close together, the cable arms are a menace. They block airflow, hold heat, and I can count the number of times I've wanted to pull a server out while it was still on at 0. They're completely worthless to me, so I don't install them. I'll leave the decision to you, but my advice is that they make excellent ballast for empty garbage cans.

Cable length makes up a big part of keeping a rack clean, too. Assuming you have a typical 42u rack, the farthest apart two network ports can be is just over 6ft. Assuming you want to square off the corners and run the cable in straight lines, it's just over 9ft. Thus, you should never have a cable on your rack longer than 10ft. If you place your switches intelligently (in this case, facing the back of the rack, and in the middle, at around 21u from the bottom), you can cut that in half, for network cables. The majority of bad cabling jobs are due to improper lengths being used. If you're making your own cables, this is easy to remedy. If you're ordering them, you'll have to plan ahead. For my blade systems, I planned the rack layout ahead of time, and ordered the exact ethernet and fiber lengths I needed. Now there's no extra cables to block air, and there's no pile of cables at the bottom of the rack that I have to worry about.

You can crimp your own ethernet, but making power cables is a little more difficult. I'd instead recommend ordering exact lengths. You can get anywhere from 1ft to really long. Chances are very good that the right length exists for you already. If you don't have the time to wait on the cables, the next best thing is to only unwind as much cable as you need from the twist-tied cable bundle they sent with the computer. When you get the right length out, twist-tie the bundle back together. It's still neat, and stays out of the way.

Vertical / Horizontal Cable Runs

You won't be running cables to servers all the time. In many cases, you'll be wanting to get the cables to a distribution panel, between patch panels, or between a panel and a switch. In these cases, you're going to be using vertical and/or horizontal cabling. The names "vertical" and "horizontal" come from network architecture ideas. Vertical cables are backbones that go between distribution facilities. Horizontal cables go from distribution facilities to endpoints.


For our purposes today, the differences don't matter much. The important part is that we're running semi-permanent cables between places. Because of this difference, where I recommended velcro before, I recommend zip ties now. Due to the number of cables that are frequently run in this way, the bundle is often too big for velcro anyway, and since the install is permanent, you won't have to worry about taking the cables down often (at least, you hope not).

Desktops

Let us not overlook the most common type of computer in offices, the humble desktop. Pound for pound, it can create a larger mess of cables than that 6u beast in the rack. Couple this with the tendency to put desktops on the floor, right next to feet, and you've got a mess in the making.

The best tools for these I've found are the plastic tubes with slits cut in them. There are some more extreme measures, but the plastic tubes usually work pretty well. The only issue is that my peripherals come with an irritatingly short cord. Most of the time, it seems like part of the cable is being wasted going to the ground, then going all the way back up to my desk. You can remove that slack with sticky cable tie squares.

Review

As you can see, there's no end-all be-all solution to cable management. It's an ongoing struggle, and the number of solutions that people have provided is mind-boggling. That just goes to show that it's a difficult problem to solve.

As an admin, there's really no book or course that you can take (that I've heard of, anyway) to improve your skills. It's very much an apprentice type knowledge, and your cable management skills are likely a reflection of who you learned from, and what you've managed to find on the internet. I know I've come by my knowledge through trying things, testing them, and using critical thinking, and I'm sure you have too. For this reason, in order to increase the general knowledge on the subject, please comment below with your own tips, techniques, and criticism of my take on it. I'm the first to admit that I'm not the best at cable management, but I've been doing it long enough that they're a lot better than they used to be. Hopefully they'll help someone earlier on the curve get better faster, and that will help us all improve.

(photos courtesy of Jonathan Auer, cableorganizer.com, madferrett, Jason Froemming, and Rob Dudley)

Excellent Linux Command: dmidecode

I can't believe I didn't know about this command. I mean, it's so simple, but it reveals SO much information.

As root, run "dmidecode" and pipe it to the pager of your choice, like this:

root@newcastle:/home/bandman# dmidecode | more
# dmidecode 2.9
SMBIOS 2.3 present.
72 structures occupying 2461 bytes.
Table at 0x000F0450.
--snip--

If you scroll down just a little ways, you start getting better information:

BIOS Information                
        Vendor: Dell Inc.                
        Version: A09            
        Release Date: 06/22/2005
        Address: 0xF0000        
        Runtime Size: 64 kB     
        ROM Size: 512 kB        
        Characteristics:    

There's more:

Base Board Information
        Manufacturer: Dell Inc.          
        Product Name: 0M3918
        Version:    
        Serial Number: ..CN7082154I04CU.

Handle 0x0300, DMI type 3, 13 bytes
Chassis Information
        Manufacturer: Dell Inc.                
        Type: Mini Tower
        Lock: Not Present
        Version: Not Specified
        Serial Number: CPJLT71
        Asset Tag:           
        Boot-up State: Safe
        Power Supply State: Safe
        Thermal State: Safe
        Security Status: None

You have to look at it yourself to believe all the information it provides. Never again will I wonder what the model number on one of my servers is:

[root@a-fs2 ~]# dmidecode --type 1
# dmidecode 2.7
SMBIOS 2.4 present.

Handle 0x0100, DMI type 1, 27 bytes.
System Information
        Manufacturer: Dell Inc.
        Product Name: PowerEdge 1955
        Version: Not Specified
        Serial Number: JJN2LF1
        UUID: 44454C4C-4A00-104E-8032-CAC04F4C4631
        Wake-up Type: Power Switch
        SKU Number: Not Specified
        Family: Not Specified

Read the manpage on it. I'm going to get right on a script to go pick up the information from remote servers and present it in a meaningful way. That should be an excellent way to provide information for the wiki


[EDIT]

As Brandon mentioned in the comments, "hwinfo" is another neat command that doesn't get installed by default (at least on my Ubuntu/RedHat systems), but definitely seems worth the time to get.

More on Admin Responsiblity

Since the drama is over, I suppose I can finally touch on San Francisco's recent network issues, namely a network admin holding the network devices "hostage". Sort of.

If you aren't familiar with the story, here's a brief rundown.

Network admin Terry Childs built the San Francisco FiberWAN, the backbone that municipal data travels on. To say he was an insanely-protective admin is an insult to the insanely-protective admin community. According to one report, he was so secretive that he refused to write configs to flash. Now THAT, my friends, is being too paranoid. In the end, the city tried to fire him, and he refused to hand over the authentication information, and he booby-trapped the network so that he could disable it and erase the config from outside if necessary. The mayor of San Francisco eventually talked him from his proverbial ledge and coaxed the passwords out of him, as apparently the mayor was the only man Childs trusted.

The end result was that San Francisco went a week and a half without having access to their WAN equipment. Now, I imagine the remaining admins are scouring every line of configuration trying to make sure that Terry didn't leave any other backdoors or vulnerabilities. I don't envy their job at all.

You've probably realized how this relates to people like us, who admin small networks by ourselves. The major mistake San Francisco made was placing one (apparently unstable) person in charge of the infrastructure with no oversight. That sounds almost like my job.

We're in this position by design. By being the only admin of a small infrastructure, we have a high bus factor. Unnecessary secrecy has no business on our networks. We touched on this last month, after MSNBC reported on IT worker ethics. Nothing has changed since then. Being prepared as an organization means guarding against employees who get hit by buses or turn evil.

NAT: Unjustly maligned or a soon-to-be anachronism?

I received my copy of Network World today (which has a free subscription if you're a sysadmin, and is worth the walk to the mailbox to pick it up), and one of the front page headlines read "Slow move to IPv6 giving NAT a new life". Apparently I've missed a couple of things.

The first thing I mentally noted was that they expected IPv6 to be moving fast enough to be a widely used, and generally standard technology. I guess from my sheltered position, it doesn't seem as impending to me. A brief survey of some associates of mine indicated a solid 0/10 where an internal IT worker (in many cases, network oriented) was aware of a plan to utilize IPv6. The companies included one of the nation's largest healthcare providers, one of the largest airlines in the world, and the 2nd largest telecommunications provider in a state, among several other SMBs. I'm going to go ahead and say IPv6 isn't something that needs to go in my calendar book yet, though I'm keeping an ear open for news of the development.

The second tidbit from that headline is that, apparently, Network Address Translation (NAT), needs a new life. As in, it's currently dead. Maybe it doesn't want to go on the cart.

Either way, and this might just be an old-fogey way of thinking, but I don't want my internal servers to be publicly addressable. Really. Not even a little bit. I don't care if they still have to go through a firewall and router. I like the fact that, due to my networking scheme, even a misconfigured firewall will prevent direct access to the internal machines through the use of NAT.

How about you? Is your organization looking at IPv6? Do you use NAT?

Read the article and let me know what your take on this is.

Escape from NJ

Well, last week was a blast. I'm only being partially sarcastic. I first want to thank my guest bloggers, Jim and Shaun. I hoped you found their articles interesting, and they might be blogging in again from time to time. I'm sorry I didn't get to write even an article, but I was so insanely busy that I ended up crashing at the hotel really late, only to end up getting up early to go back into the office.

On the subject of work, let me first recommend to you: If you decide to go with a colocation, and that colocation offers a service where they will do a one time rack install for you, do it. Don't even question it. Since I was the only one there from my company, I saved myself easily two, maybe three days worth of work, and several hernias. Our equipment was professionally racked and cabled, and everything was done according to my specifications. The most I had to do was show the workers my diagrams of where I wanted things to go. I really spent 3 hours moving empty boxes out of their way and queuing up the next round of equipment. It was great. I can't recommend it enough.

I also managed to get the T1 at the corporate office tested and turned on. I mentioned a while back that I had DSL at the corporate office, and that I was unhappy with it. Getting the T1 turned on is a big change for them.

Now, I'm using the Juniper Netscreens to route all the VPN traffic through the T1, and all the normal web-browsing traffic through the DSL (with the T1 as the fallback for normal traffic, and the DSL as the fallback to the VPN). This complex setup took me about 14 hours to get working correctly. I'm glad it's done.

The actual being-in-NJ part was nice. I like my office in Ohio, but since it's only me and my boss, it's quiet...almost TOO quiet. That's definitely not the case in the corporate office. There's also the fact that my coworkers there don't always tell me something is broken. They save up lists and have me fix things en masse. I'd much rather they submit the issues in bugzilla like I've asked them to. At least they tell me things before they catch on fire.

Anyway, thank you all, and I'll be back to my normal blogging schedule (if you can call it that) this week.

Wednesday Guest Blog

Today's guest blog entry is from Shaun Hurley and an errant certificate. Enjoy

A Lesson in SSL Troubleshooting

It was an odd problem.

For some reason our trusted business partner (BP) was unable to send an SSL certificate when they made a call to our IIS web-server. The response from the webserver: 403.12 - Client certificate has been denied access by the server certificate mapper. The connection was based on a client side certificate, verified against the public key on our end that they provided for us. When we tested it with our certificate from a remove location it was OK (should have been a red flag).

Our BP wanted to know what was in the path from their server to our server. Well, the server they were hitting was in our development lab, so: a border firewall, several routers, and another firewall to be able to access our lab. They wanted to know if one of our devices (firewall, in particular) was stripping out the SSL certificate as it was being sent across the network. Short answer: no.

Perhaps someone can correct me if I am wrong, but if 443 and 80 are both open, valid (sometimes invalid) traffic should be allowed to cross the firewall. Now there are other small

details: such as valid session states, etc. If there were a problem at the firewall level, we wouldn’t be able to hit the server. This shouldn’t have been called into question.

Besides, the error message indicates a problem with either the server or client certificate. They connected using openssl, it displayed a list of options for Certificate Authorities (CAs) and didn’t see the one there for the CA they are using for their certificate. So they sent it to us, claiming that

it was and old Verisign CA that is apparently not on our trusted server list (another flag).

So they sent it to us, we imported it, and now the CA was in the list but they were getting a different error. 401.13: client certificate has been revoked on the web server. Yay! Something new.

So we played around with the IIS settings, turned off all certificate checking to verify if it is a valid certificate (big no-no). And we finally got a new error message: 401.16 – client certificate

is ill-formed or is not trusted by the web server. Now we are getting somewhere.

We asked them about the validity of the certificate, but they claimed that it works with other clients, but my co-worker (thank god), a meticulous worker, forced them to compare every piece of their public-key with the private key they have generated. Things were matching up, until we found out that not only was this certificate NOT generated by Verisign, it had the wrong hashing algorithm. On top of that, we found out that the CA they sent us was NOT an old Verisign CA. Removing the CA, and changing the settings back, we had them generate a valid certificate.

All is well.



Lessons:

Always check the minor things. It never hurts to double check the validity of a certificate; no matter how
certain you are that it is OK. Small things like this can prevent a couple days of banging your head against
a wall trying to figure out the problem.


Shaun Hurley

Shamrockhoax.blogspot.com

Myspace.com/shamrockhoax

A facebook thing-a-ma-bob as well.

Something to Ask: If anybody know of a virus that will infect juke boxes that will remove the band OAR from the machine, I would be eternally grateful. It is time for their horrible brand of soft-mold, half-baked, [expletive deleted] music to go the [expletive deleted] away. I swear to god, if I hear that steaming turd of a song called “crazy game of poker” one more time, I will shoot somebody (Nickleback is next, I promise). 30 minute songs about some hippy’s lame acid trip/white boy tribute to jah should not be a song/should not exist on a jukebox. EOM.

Monday Guest Blog

Today's guest blogger is Jim Wells. Enjoy!

For those regular readers of the blog, I'm the junior admin mentioned here. I'm currently at a stop point on my current project (new PDC), so I figured I would start updating our internal wiki. So far I've found that while most of the wiki information is current, it is also somewhat... brief.
Yesterday (7/10), I decided to document our static IPs internal and external. I know there is probably a right way to go about this, but being I didn't know that way, I went with my way. I ran a basic port scan on the internal network to find out what I actually have online, and I also dumped the internal DNS zone to see what it had. While I didn't find anything on the network that wasn't supposed to be there, I did find some old clutter in our DNS server. And by old, I mean references to systems/projects that were long dead before I started working here two years ago. So that got pruned and into the wiki.
Also on my list of recent updates was our server listing. After the initial conversation with Matt, I went through and updated our system list to make sure all the servers were listed along with a rudimentary description of what they are for. Today's work will involve collection Service Tag numbers and system specs along with a detailed description of what services they are running. If I'm lucky at pinning down the boss, I also want to see about getting a replacement timeline set up. With the funding we get around here, it'll be more pipedream than actuality, but it would be nice to have something down on paper.
What I would like to ask the rest of you reading out there is this: What else (idealy) should be in my network documentation? I'm still pretty new to this line of work and I am not sure what I really need to have down. It's hard to pin down my boss to get this information sometimes, and I would like a broader prospective on the issue. I look forward to your replies.

Jim

(Short) Story time

I forgot to mention this, but today I found this excellent short story by Cory Doctorow. It's called When Sysadmins Ruled the Earth.

It's definitely an interesting read, and the jargon is refreshingly accurate. This was the first story I've read by Mr Doctorow (real website here). I'm going to have to read some more of his, definitely.

Does anyone have any favorites or suggestions?

Final Preperations

We had our final conference call today with the colocation representatives. It turns out that we're going to need a 30 amp circuit rather than the 20 amp line they installed. Of course, we knew this since the beginning, but there was a typo in the spreadsheet that got sent to the salesman.

The end result of this is that we're not going to have the circuit in place for the Tuesday install. Thankfully, the 208v circuit is in place, and we can use the 20amp circuit until the 30 amp is installed. That'll let me make some progress. I've just got to provide very thorough instructions for migrating to the power source.

I came in early today and finished some network diagrams and paperwork that ensured our equipment's entry to the colocation. This is our 2nd colocation, but it's the first time we've had to do anything requiring this level of detail. As soon as I get some time, I'm going to be writing up tips to help someone else. This has been eye-opening. The next time will be a lot easier. I told my boss today, it's a shame that the next time we do this, we'll be installing the backup, because it's probably going to be a lot smoother.

As an eye-candy bonus, here's the layout of the rack I'm installing. The only additions that aren't on here are the PDUs and the KVM for the two non-blade machines.


(click for the flickr page)

Anything in the rack that has ethernet ports on the front will be turned around and mounted in the rear. I'd prefer to put the blades a little higher, but of all things, cable length is dictating this placement. It works this way pretty well, and it gives me a wide open space at the top if I need to expand.


As an aside, since I'm going to be out of touch for most of next week, I'm planning on having some friends of mine filling in and doing guest spots. I haven't received any submissions yet, so I can't give you any hints about what they'll cover. It'll be as exciting for me as it is for you, I'm sure :-)

Take care and wish me luck!

Cataloging parts for the move

One of the many bits of detail the colocation we're moving into wants to have on file are serial numbers for all the equipment. That's fine, and I need those numbers for my own record keeping. I figured while I was going around, I would double check that everything was labeled and accounted for.

I use a Brother P-Touch, like this one:


I like this over a lot of other models, because the tape backing is split down the middle, as opposed to having to try to bend a corner and get your fingernail between the sticker and the backing. This is much easier.

The one bad thing is that it wastes a lot of tape. I mean, a lot. There's a good inch on each side of the printed text where the tape is completely unused.

To get around this, I always make sure to print multiple labels at the same time. If I'm printing the lables "bs-1001" through "bs-1010", I type them all at once, with a couple of spaces in-between. This eliminates the 2 inches of wasted space. I just cut them apart with scissors before I peel the backing off.

You've got to be careful in the server room, because these little bits of paper float away. Or get sucked into the fan intake on the front of rackmount cases.

Anyway, I'm going to be labeling all the ethernet cables for the switches. Thanks to the blade form factor, there aren't many power cables, but the ones I do have are going to get labeled. The front and back of the two rackmount servers are getting labeled with the assigned internal part numbers to track them, and so will the other various bits that have escaped my efforts thus far.

It's important to document this while you're doing it, too. In this case, especially, since the on-site colo personnel is going to be doing the actual install. In my mind, I'll have a lounge chair somewhere near the server rack, and a tropical drink complete with little umbrella in my hand, while I occasionally bark orders at the peons who have to lug and lift the equipment. Somehow I doubt real life will be remotely like this.

Unless I provide those people with a proper map on how to install my stuff, it's hard to tell what I'll end up with. In the future, I hope to have another admin to work with, and when I send them to the colo to do work, I'd like them to be a little bit prepared for what he's up against.

I'm sure there's a great way to diagram cables, but I don't have enough experience with Visio yet to use that, and to be completely honest, the line connections on OmniGraffle are miserable. Anyone have an idiots-guide-to-visio suggestion?

Here's the link to the Brother labeler at Tiger Direct. I figured I should link to them since I borrowed their picture.

Stupid Routing Tricks

As you might have read, I'm just about done building the new equipment stack that is going to go into the new co-location. I'm at the stage where I'm trying to finalize all the equipment settings. This includes pesky things like setting up the external firewall cluster with the right IP configuration.

There is the small difficulty that my external devices have a public IP block that isn't routed to my office. Effectively, this means that I've got to do bizarre, black magic to get my internal equipment to recognize the a.b.c.0/27 network our new colo provided us. I've done it, and I feel dirty. Here's a diagram of essentially what I've got now:


The networking-oriented among you will probably see the issue immediately. With the strategic use of static routes, I can get our internal machines to see the external interfaces of the colo netscreens, and once that happens, I can setup the VPN tunnel between the two. However, at this point, there's no way short of NAT to get the colo machines access to the internet. As soon as traffic leaves my 2611, the source IP gets seen as the colo-provider's IP, and routed far, far away.

I can use NAT, but then I'll have to change the config on my 5GTs, which I'm hoping to avoid. Today I may try some static route trickery to have the colo machines temporarily route everything through the VPN tunnel and let the 5GT's take care of traffic. That may work.

Anyway, if I come up with an interesting solution, I'll be sure to let you know!

The effects of Slashdot

If you're still visiting the blog from Slashdot, hello!

I submitted the article DNS Names for Internal Hosts to Slashdot, about a week ago.

Yesterday, they ran the story.


I didn't get any sort of immediate notice. It was Sunday, and I don't usually check Slashdot on Sundays. I was actually sitting on my laptop reading an E-book when I got a notice from my GMail notifier that someone commented on the story. I didn't think anything of it, and I kept reading. Someone else commented.

Just so you know, I'm not what I would consider a "high traffic blog". I have around 30-40 regular readers. There's a link to the blog in my slashdot signature, so when I comment on stories, I usually receive a spike in traffic from people clicking the link. I checked a while back, out of curiosity, and I get around 20 visitors for every comment I leave.

Anyway, my average traffic is 30-40 visits per day. Yesterday, my graphs got thrown off a little bit:



It's a good thing I don't host this blog on a server that charges for bandwidth.

And remember, this was on a Sunday evening. The story garnered over 350 comments on slashdot by this morning, and I got 37 comments on the blog.

In terms of pageviews, out of all those people, only 225 looked at the main page after reading the blog entry. 50 more went to previous pages.

I write this because the macro-trends of visitors interest me. It's neat to try to understand why some people click links and other people don't, or what draws visitors.

If you're at all interested in who visits your site, and from where, and what they do when they're on it, I cannot recommend Google Analytics enough. The amount of information is mind-boggling, but it's well presented and easy to use. We've come a long way from old-style counters.

Remote Claustrophobia

This is one of those little whining complaints, but I really don't like like working remotely. I have a certain interface at work that I'm used to, and when I work on my dinky-feeling 15" powerbook, I miss the other 25 diagonal inches.

I've got two 20" widescreen monitors as my desktop at work, running Ubuntu, with WindowMaker as the window manager. It's configured primarily for speed, with an emphasis on bringing terminal windows up, and quickly. On a typical day when I'm working on several projects, it's typical to have 20ish terminal windows spread over 7 desktops.

As much as I like OS X as a desktop operating system, when I try to do work on it, it just seems so...slow. Click here, click there, click over there.

Maybe there are ways to assign keyboard shortcuts to certain programs, but if so I haven't found it. Since I'm on an old powerbook, I'm still on 10.4, so by default, I have one desktop. I installed a free virtual desktop program that helps a lot, but I can't get over the feeling of being cramped and hampered.

It doesn't help that I work in a recliner when I'm at home, and I get headaches from my bad posture.

/end whining

Logical Network Diagrams

As I've often mentioned, network documentation shall set you free. It enables monitoring, it clarifies murky or misunderstood concepts, and it lends itself to allowing improvement of the infrastructure.

Ideally, documentation should be the first step of the building process. Architects don't get a bunch of lumber and throw it around until it looks cool, then draw blueprints. They plan first, and so should you.

I'm nearing the end of a build process for my stack of hardware that's going into a colocation in NJ. In terms of network diagrams, I've gone from back-of-the-napkin drawings of clouds to fully realized diagrams created in OmniGraffle (a very, very cool diagramming program if you use Mac).

Here's what the logical network at the colocation will look like:



As you might have noticed, the only thing that isn't symmetrical is the Brocade fibre-channel switch. That's because they're several thousand dollars. I tried to get the budget for it, but it was no go. I'm still looking for a cheaper solution that doesn't involve a single failure point.

Anyway, as you can see, this diagram can help me understand at a glance where any specific piece of hardware fits in the scheme. There are lots of other ways to diagram the same thing. For (a lot) more information, check this great link: http://www.networkdocumentation.com/.

DNS Names for internal hosts

Bob Plankers, over at The Lone Sysadmin wrote a couple days ago about getting busted while reading the wiki page on X-Men. He tried to cover it up by claiming to be researching future host names. Quick thinking, Bob. Good job! ;-)

It does bring up a good point, though. Internal naming schemes are something that everyone has an opinion on, and a load of suggestions.

At various places, I've used greek/roman gods, Simpsons characters, beer companies, wine labels, and fish.

At my current company, we used the beer and wine names. We absorbed another company that used fish. It worked fine for a while, but we grew in terms of servers and locations until it got unwieldy to remember A) all the names, and B) what each name did. You'd also start to get very similar names after a while. We've now got 4 physical locations, soon to be 5, and something like 50-60 servers (not counting network devices), no one would be able to keep them all straight (including the admin).

To improve the situation, we're in the process of changing to location-based hostnames with a flat internal domain structure. For example, the 2ndary application server in Ohio is oh-app2, with the fake internal domain name trailing. The alpha site's primary fileserver is a-fs1.

It's no where near as fun as "wolverine.internal.com" but it certainly does tell you where you're connecting to and what the machine does. What makes it interesting is when you go changing things like CVS repositories on people's machines, mail servers, etc. The policy we've taken is to alias the old information to the new, and slowly phase out the old method.

What do you use as internal naming systems? What do you think would make an excellent scheme? Make sure to check the list to make sure it hasn't been done before!