Showing posts with label networking. Show all posts
Showing posts with label networking. Show all posts

Friday, May 1, 2009

Random thoughts on Slashdot

I wanted to make a quick reply to someone on slashdot who suggested adding a 5th octet to IP addresses rather than migrating to IPv6. I meant to write a really quick reply, but it got drawn out. I got done with it and thought that some of you might have thoughts on it:

Awesome idea. We'll give Google 1/8, The government can 2/8, IBM will get 3/8, etc etc etc

Same problem. The ipv6 is not a "bad" idea, it's just sort of like...imagine in 1950s if the phone company decided "we could go with area codes to subdivide numbers to prevent running out, or we could use letters AND numbers".

Can you imagine the upheaval?

In a lot of ways, that would have been even easier to deal with, because everyone's phone was owned by AT&T. New phones could have been issued without too much problem.

No, imagine it instead in the mid 1980s. Ma Bell doesn't own the phones any more, in fact there are tons of cheap phones available, cell phones are starting to come out, and there are still rotary AND push button phones.

That's more like what the IPv6 switch is like. Do you give the new people 2 numbers, so that grandma can still call them? How long is it before you stop accepting legacy phones that only have 10 dialing options? How the hell do you get DTMF to work with 36 numbers? Do we need area codes? It would be weird without them, but we don't really need them.

The equivalent of these questions are still being asked. Just a couple of months ago, there was a huge to-do about NAT and IPv6. "IPv6 is a world without NAT". The hell it is. My internal routers don't get publicly routable IP addresses, even if I have to NAT back to IPv4.

When the wrinkles get ironed out, we're going to wonder how we ever did without it. During the transition, it's going to be hell for everyone (with the possible exception of the clueless end user, who might have to buy a new router at most).

Wednesday, April 22, 2009

Discussion on /. - Should cables be replaced?

Yesterday there was a pretty good discussion going on over at Slashdot regarding changing out network cables. I thought some of you might be interested in it. Sorry for the tardiness, but better late than never, I suppose.

Another thread to add to my SlashDocs bookmarks!

Tuesday, April 21, 2009

My trouble with bonded interfaces

In an effort to improve the redundancy of our network, I have all of our blade servers configured to have bonded network interfaces. Bonding the interfaces in linux means that eth0 and eth1 form together like Voltron into bond0, an interface that can be "high availability", meaning if one physical port (or the device it is plugged into) dies, the other can take over.

Because I wanted to eliminate a single point of failure, I used two switches:



The switches are tied together to make sure traffic on one switch hits the other if necessary.

Here is my problem, though: I have had an array of interesting traffic patterns from my hosts. Some times they'll have occasional intermittent loss of connectivity, sometimes they'll have regular time periods of non-connectivity (both of which I've solved by changing the bonding method), and most recently, I've had the very irritating problem of a host connecting perfectly fine to anything on the local subnet, but remote traffic experiences heavy traffic loss. To fix the problem, all I have to do is unplug one of the network cables.

I've got the machine set up in bonding mode 0. According to the documents, mode 0 is:


Round-robin policy: Transmit packets in sequential
order from the first available slave through the
last. This mode provides load balancing and fault
tolerance.



It would be at least logical if I lost 50% of the packets. Two interfaces, one malfunctioning, half the packets. But no, it's more like 70% of the packets getting lost, and I haven't managed to figure it out yet.

If you check my twitter feed for yesterday, I was whining about forgetting a jacket. This is because I was hanging out in the colocation running tests. 'tcpdump' shows that the packets are actually being sent. Only occasional responses are received, though, unless the other host is local, in which case everything is fine.

There are several hosts configured identically to this one, however this is the only one displaying this issue. Normally I'd suspect the firewall, but there isn't anything in the configuration that would single out this machine, and the arp tables check out everywhere. I'm confused, but I haven't given up yet. I'll let you know if I figure it out, and in the mean time, if you've got suggestions, I'm open to them.

Monday, February 2, 2009

(Really) Small Office Environments

Many thanks to Jeff Hengesbach for this blog entry!

There are a lot of very small businesses. I'm thinking about 20-30 or fewer people, and likely only 1 or maybe 2 servers. There are a few reason this interests me. First, in the past I've done 'side' work in a few of these environments. And secondly, every time I enter a place of business of any sort, I'm always looking for technology and how it is being used. For the folks helping these smaller organizations out, I like to scale back some 'bigger' business concepts and show how they are advantageous for everyone.

The one thing that never ceases to amaze me, when I gain knowledge of it, is the age of the oh-so-critical systems these companies rely upon. My background and philosophy on physical (x86) servers life-cycles is 3-4 years and replace. I follow this cycle for multiple reasons: 1) OEM warranties are cost effective in this window, 2) Always run warrantied equipment, 3) Computational power leaps, storage costs plummet in 3-4 years, 4) It fits a good window for OS / application upgrades and, 5) Equipment is not that expensive in these environments. Of course for large / complicated systems these arguments don't hold as much water.

Over the past few years, I've seen 2 organizations lose servers from aged hard drives and other major component failures. They thankfully both had good backups, but where still out a bunch of time during the replace and rebuild process. If you think disk mirroring / system mirroring is a backup solution, please read this article on Slashdot.org.

The direction I'm heading with this is to ask small shop IT to consider the use of virtualization. On a small scale the solutions are virtually free, pick the one your expertise best fits. Consider the cost of a down system seriously - it will happen. A Virtual Machine image can be pulled from backup(choose your media wisely), up and running on PC bought from the local big box store in very short order(depending on your VM solution). Get your replacement server, copy the VM over and the case is closed. No Windows hardware driver issues, Authoritative AD restores, configuration oversights, etc. Virtualization will also make for a near painless experience when it comes to keeping physical servers upgraded.

If your small office IT support isn't up to speed with Virtualization, ask them to get there or find new help. The benefits are too great and easy to reap to let them pass by.

Friday, January 30, 2009

Redundant Redundancy

Many thanks to Ian Carder for this blog entry!

Over the past year we have replaced all of our remaining Cabletron network equipment with Cisco gear. Being a K-12 School District, we have to be extremely frugal where we spend out money. Often, this comes in direct conflict with going the extra mile to make sure whatever system we're deploying has some redundancy in the event of a failure.

Sometimes I can “sneak” redundancy in without having to pay a hefty price or fight for it. This involves simple things like RAID 1/5/10 on servers, dual power supplies, and dual supervisors in the Cisco core switches. We have been making heavy use of ether channel from MDFs to remote closets. That's a cheap solution assuming you have the available ports and don't feel like you have to over pay for Cisco branded GBICs. The other project we're looking to wrap up is creating more that one route to each building in the district using at least a hub network topology. Sorry, no spokes yet! The benefit here is that if we lose a single building, not all traffic is cut off between buildings on either side of the problem building. Lucky for us, we're all on one LAN, so no shared bandwidth to deal with. Just as with the ether channel situation, as long as you have the cabling between sites and available switch ports, it's a cheap proposition. I also have a small cluster running Novell's Cluster Services, but that was something I had to fight for. Beyond that, everything we have is single tier.

So I pose a couple of questions to the faithful readers; what critical services do you need to make redundant and what on the cheap tricks have you come up with to provide the redundancy?

Wednesday, January 28, 2009

vnstat Console Network Traffic Monitor

Many thanks to Nick Anderson for this blog entry!


Matt was kind enough to ask me to spice things up a bit while he is gone on vacation.(well kind to me, perhaps not so kind to his readers).

If you have ever wanted to monitor network bandwidth I'm sure your familiar with mrtg. I find it useful to be able to do things from the console whenever possible. vnstat fills the void for simple bandwidth monitoring. Since vnstat pulls its information from the /proc filesystem it can be used without root permissions.

To install vnstat on debian just do

[code=bash]
sudo aptitude install vnstat
[/code]

Now you need to create the databases for it to log to. For each interface you want to monitor do the following.

[code=bash]
sudo vnstat -u -i iface
[/code]

After you have created the databases for each interface you want to monitor check to ensure the cron entry was setup correctly. You should have a file in /etc/cron.d called vnstat and its contents should be similar to the following.

[code=bash]
# /etc/cron.d/vnstat: crontab entries for the vnstat package

0-55/5 * * * * root if [ -x /usr/bin/vnstat ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then /usr/bin/vnstat -u; fi
[/code]

Now all you have to do is wait. After some time you can get interesting output like this.


[code=bash]
vnstat --days -i wlan0

wlan0 / daily

day rx | tx | total
------------------------+-------------+----------------------------------------
05.01. 9.69 MB | 1.84 MB | 11.53 MB
06.01. 410.54 MB | 23.72 MB | 434.26 MB %
07.01. 409.89 MB | 25.83 MB | 435.71 MB %
08.01. 68.63 MB | 12.91 MB | 81.54 MB
09.01. 9.75 MB | 3.73 MB | 13.49 MB
10.01. 292.16 MB | 39.78 MB | 331.94 MB %
11.01. 178.23 MB | 1.57 GB | 1.74 GB %:::::
12.01. 350.97 MB | 1.60 GB | 1.94 GB %:::::
13.01. 23.65 MB | 5.89 MB | 29.54 MB
14.01. 410.56 MB | 1.61 GB | 2.01 GB %::::::
15.01. 440.17 MB | 1.62 GB | 2.05 GB %::::::
16.01. 241.75 MB | 26.32 MB | 268.07 MB
17.01. 713.33 MB | 79.22 MB | 792.56 MB %%
18.01. 5.40 GB | 1.64 GB | 7.05 GB %%%%%%%%%%%%%%%%%%%::::::
19.01. 168.10 MB | 1.45 GB | 1.61 GB %::::
20.01. 19.27 MB | 9.64 MB | 28.91 MB
------------------------+-------------+----------------------------------------
estimated 21 MB | 10 MB | 31 MB

vnstat --months -i wlan0

wlan0 / monthly

month rx | tx | total
-------------------------+--------------+--------------------------------------
Jan '09 9.06 GB | 9.72 GB | 18.78 GB %%%%%%%%%%%:::::::::::
-------------------------+--------------+--------------------------------------
estimated 14.13 GB | 15.16 GB | 29.30 GB
[/code]

Hopefully its something you can add to your command line toolbox.

Monday, January 26, 2009

My Most Oft Used Cisco IOS Commands

Many thanks to Ian Carder for this blog entry!

Over the past year we have replaced all of our remaining Cabletron network equipment with Cisco gear. Being a K-12 School District, we made use of Cisco Catalyst Express series of switches in our IDFs. The CE series of switches are configured using a web browser. We do have a large install base of Catalyst 2950s and 3550s. Our core network switches, Catalyst 4500 series, are all configured on the command line using IOS. The network configuration is in place and not changing any time soon. Most of the IOS work I do now are simple tasks such as changing switch-port VLAN designations and diagnostics. Here are some of the commands I use most often:

do show run
I always find myself needing to double check something in the running config while I'm in configuration mode. Instead of dumping back into exec mode, issuing do sh run will enable you to check your running config. In general, the do command is to run any exec mode command while in configuration mode.

show run | begin
When you have to check how you have an interface configured on a loaded Catalyst 4500, I find myself hitting space bar a lot and half the time I blow right by the interface or line I was looking for anyway. Assuming I know what I'm looking for and it's a unique entry in the running config, piping the begin command with whatever text you're searching for after a sh run will jump the sh run output right to the text you specified after the begin command. An example I might do would be a sh run | being 6/26 to see the config for blade 6/port 26. You can also combine it with the do command if you're in configuration mode.

show ip route
Generally I don't have to check out the routing table. Our network is fairly stable at this point. Anytime I have to add a new subnet, I'll end up issuing the sh ip route command just to be sure that the subnet has 'propagated' throughout the network. If you're doing more layer 3 work in your network, you'll probably be using the command often.

show vlan
I use this command to quickly see if an access or voice vlan is up, and to see what ports are associated with it. If you create a vlan on a switch, it won't automatically become active unless you assign an interface to it.

show interface
The show interface command is extremely useful for diagnostics. There is a ton of useful information for each interface that you run this command against. The most basic output is whether the interface is up or down, and whether you shut it down(administratively down) or not.


If you find yourself schlepping around IOS often, or any other platform for that matter, feel free to talk about commands you use most often and why.

Tuesday, September 2, 2008

Random thought on iSCSI vs FC

I read Jeff Hengesbach's blog every day, and I'm fairly sure that if you like my blog, you'll love his. He recently started it, and every post so far has been really interesting. You should check it out.

Anyway, today Jeff wrote about iSCSI SANs. If you're unfamiliar with the concept, here is a great introduction. You should also know that there are two competing standards, iSCSI (where the storage is accessed over standard ethernet) and Fibre Channel (where the storage is accessed over fiber optics cables). There's some more in-depth explanation here, but it all boils down to speed and expense.

Among the in-the-know people (AKA: not me), it seems to be a sort of common knowledge that we're going to be using iSCSI in the future, what with 10Gb ethernet being the standard just-around-the-corner. The current best-of-breed is 8Gb/s FibreChannel from IBM, but 10Gb would trump that, and the added bonus of using tried and true ethernet is very appealing to a lot of people.

Regardless of whether or not we end up with iSCSI I'm not convinced that copper will last us much longer in the grand scheme of things. I suspect that, maybe by the time 100Gb rolls around, we'll be using ethernet-over-fiber for that stuff. The theoretical bandwidth of optical is just too high to ignore in the long-term. If you want to get all sci-fi, scientists have recently began working on entirely optical versions of most basic integrated circuits.


Just a random observation to spark discussion. What do you see happening in the future of SANs?

Saturday, August 30, 2008

A look at undersea cables

Browsing through Digg today, I chanced across this article about Google expanding their undersea cable portfolio. Aside from increasing the inter-Asian bandwidth by over 7Tb/s, it will be Google's 2nd undersea cable, the first being the transPacific cable known as "Unity". As recently as a year ago, the first cable was just a rumor. In February, they confirmed that it would be in place and operational by 2010. The first article I linked to also mentions that Google is looking at an African fiber to replace the aging SAT-3 cable. Offhand, to give you some idea of the magnitude of the new fiber, SAT-3's bandwidth is a paltry 130Gb/s.

What does this have to do with small systems administration? Not much specifically, to be honest, but it's interesting to me. I look at the global fiber network as a maybe a macrocosm of my own network. I look at the global fiber maps and note the critical junctures, the (sometimes lack of) redundant bandwidth in certain corners of the world.

If you're interested in this as well, here are some interesting links that I found.

Eyeball-series.org has a couple of pages dedicated to mapping the exact spots where each of the trans-Atlantic and trans-Pacific cable landings. This is a very, very bandwidth intensive page. Far more interesting are the maps at the top of each page that shows the logical circuits. Here, I'll save you time and bandwidth: Atlantic and Pacific.

This map from CNET gives you some idea of the intercontinental bandwidth available around the globe.

Here is a series of maps showing the history of undersea cables.

If you're curious as to how many cables are out there, Wikipedia has a page dedicated to listing all the cables in the individual bodies of water. The main page on submarine communications cables has a great schematic of the composition of undersea cables. How cool does this look:

Tuesday, August 26, 2008

5 Ways to improve your network without breaking the bank

In reality, there are far more than 5 ways to easily improve your network's security and reliability, without spending hardly any money at all. I've just briefly gone over the most blatant 5 that make the biggest difference for next to no expenditure. Sure, a few office supplies might need to be sacrificed for the good of the many, and you will have to change the way you approach some problems, but read this list with an open mind. You might find it more efficient than the things you've been doing.

5) Take care of your networking cables
Nothing will kill packets with more random violence than an ethernet cable that hasn't been taken care of. Whether the culprit is a loose end, bent wires from folding it back on itself, or accidentally cutting it when you were removing plastic zip ties, the end result is that your quality degrades and packets get dropped. This can be prevented by recrimping the ends when necessary, or replacing the whole cable if need be.

4) Set your wireless power level
You probably don't work in an area that covers 100,000 square feet of space, which means that lots of the wireless signal you're sending out is being wasted. More troubling, it's much easier for the casual war driver to pick them up from the parking lot (or adjacent floors). There's just no need. Most APs support adjusting the power output from the radio. Lower the signal until you can't get it in the farthest reaches of your office, then bump it back up until you can get it again. It won't defeat someone dedicated who uses a pringles can, but it's better than nothing. Also, make sure you use WPA.

3) Optimize DNS
If I were to wildly make up numbers, I'd say that 60% of weird network errors can actually be traced back to DNS, particularly if your infrastructure has multiple DNS servers which are all manually edited then updated. It's easy to fatfinger a line in the config file, or forget to update all the servers but one, or to leave out a reverse DNS entry for a host, and with the right (or wrong) host configurations, each of these can cause seemingly random havoc on unsuspecting users. "Why is it taking to long to ssh into BoxA?". It might be a network issue, it could be a server problem, or, in my case, the reverse DNS entry COULD have an extra : in it. Solve these kinds of problems by scripting DNS changes. Do less by hand and eliminate accidental screw ups. That way you can concentrate on all the bad things which you do on purpose. This entire process of automating changes and then documenting them is called "Change Management". Look into it, it's a Good Thing(tm)

2) Monitor Bandwidth
If you've got managed switches, there's no excuse to be wasting those expensive configuration and reporting abilities. Pretty much every networking device and almost every network host has the ability to be monitored using SNMP. To ignore this potential source of pretty graphs is foolish. Get MRTG, or even better, Cacti, and get to work. Knowledge is power, and learning that what should be a small filesync is actually backing up a user's entire iPod directory, every night, helps you in many ways. Learning discretion by not hitting them with a foam bat, for one.

1) For the love of God, document!
If it's something that you could concievibly have to do again, and it took you longer than 10 seconds to figure out, document it. Document it someplace that makes sense. Put enough detail in that you can recreate your actions, or better yet, someone else can recreate your actions. The last thing you want is to get a promotion to a corner office and then have some peon come bothering you about network configurations while you should be lighting cigars with $100 bills. Document properly and this won't happen. I promise.


Thanks for listening. There are many, many simple ways of improving your network. Post some of your favorites below. If you liked the article, make sure to Digg it

Wednesday, August 20, 2008

Really bad cabling jobs!

I found this amusing/depressing series of pages involving some epic wiring jobs

http://www.darkroastedblend.com/2007/03/really-bad-wiring-jobs_20.html

God help you if you're stuck with anything remotely this bad.

Friday, August 15, 2008

OpenOffice trick to auto-increment IP addresses

I've been fleshing out my IP address spreadsheets, and I've been using a trick that I didn't know about for a long time, till someone on a forum showed me. It was indispensable.

You know how, if you want a series of numbers down a row, you can click the lower right hand corner, and drag it down, and it autoincrements the numbers for you? Well, in oocalc, if you try that with IP addresses, it just copies the original IP address. I don't know why, but it does. MS Office gets it right, heck I think even K Office will do it. OpenOffice, not so much.

The trick to doing it is to put a space in front of the IP address, first.
" 192.168.0.1" will allow you to drag it down as far as you want, incrementing the last octet.

Wednesday, August 6, 2008