Introduction to LVM in Linux

I'm going to go over the Logical Volume Manager (LVM) today, because it plays an important part in next Fridays' howto. I'm not going to let the cat out of the bag, but just know that it's been a long time coming. There are lots of resources for the technical management of LVMs, and they cover the many uses and flags of the commands far better than I could here, so some resources will be listed after the entry. I'm just going to concentrate on the "why" of LVM. I've often found that learning why you want to do something is more difficult than the technical process by which you accomplish it.

First off, lets discuss life without LVM. Back in the bad old days, you had a hard drive. This hard drive could have partitions. You could install file systems on these partitions, and then use those filesystems. Uphill both ways. It looked a lot like this:



(click for full size)

You've got the actual drive, in this case sda. On that drive are two partitions, sda1 and sda2. There is also some unused free space. Each of the partitions has a filesystem on it, which is mounted. The actual filesystem type is arbitrary. You could call it ext3, reiserfs, or what have you. The important thing to note is that there is a direct one-to-one corrolation between disk partitions and possible file systems.

Lets add some logical volume management that recreates the exact same structure:



(click for full size)
Now, you see the same partitions, however there is a layer above the partitions called a "Volume Group", literally a group of volumes, in this case disk partitions. It might be acceptible to think of this as a sort of virtual disk that you can partition up. Since we're matching our previous configuration exactly, you don't get to see the strengths of the system yet. You might notice that above the volume group, we have created logical volumes, which might be thought of as virtual partitions, and it is upon these that we build our file systems.

Lets see what happens when we add more than one physical volume:




(click for full size)

Here we have three physical disks, sda, sdb, and sdc. Each of the first two disks has one partition taking up the entire spae. The last, sdc, has one partition taking up half of the disk, with half remaining unpartitioned free space.

We can see the volume group above that which includes all of the currently available volumes. Here lies one of the biggest selling points. You can build a logical partition as big as the sum of your disks. In many ways, this is similar to how RAID level 0 works, except there's no striping at all. Data is written for the most part linearly. If you need redundancy or the performance increases that RAID provides, make sure to put your logical volumes on top of the RAID arrays. RAID slices work exactly like physical disks here.

Now, we have this volume group which takes up 2 and 1/2 disks. It has been carved into two logical volumes, the first of which is larger than any one of the disks. The logical volumes don't care how big the actual physical disks are, since all they see is that they're carved out of myVolumeGroup01. This layer of abstraction is important, as we shall see.

What happens if we decide that we need the unused space, because we've added more users?

Normally we'd be in for some grief if we used the one to one mapping, but with logical volumes, here's what we can do:




(click for full size)
Here we've taken the previously free space on /dev/sdc and created /dev/sdc2. Then we added that to the list of volumes that comprise myVolumeGroup01. Once that was done, we were free to expand either of the logical volumes as necessary. Since we added users, we grew myLogicalVolume2. At that point, as long as the filesystem /home supported it, we were free to grow it to fill the extra space. All because we abstracted our storage from the physical disks that it lives on.

Alright, that covers the basic why of Logical Volume Management. Since I'm sure you're itching to learn more about how to prepare and build your own systems, here are some excellent resources to get you started:

http://www.pma.caltech.edu/~laurence/Linux/lvm.html
http://www.freeos.com/articles/3921/
http://www.linuxdevcenter.com/pub/a/linux/2006/04/27/managing-disk-space-with-lvm.html

As always, if you have any questions about the topic, please comment on the story, and I'll try to help you out or point you in the right direction

Temporary Ignorance

It's an interesting perspective, looking back and realizing how long you've spent to obtain the knowledge that you have. It's hard for me, because I think about how much time I spent chasing the wrong angles, or looking in the wrong places.

I spent at least two months trying to get RedHat Cluster Suite running on Fedora Core 6. Heck, I spent months setting up Fedora Core 6, because "It's what RedHat Enterprise Linux 5 is based on". Nevermind that I didn't understand that CentOS was a 1:1 mapping of RHEL, and that FC6 doesn't have support any more. If only I'd have had that small, tiny bit of knowledge, I wouldn't have all-but wasted months of my time.

Here's another example. I've been working on trying to get my Dell blades to their maximum availability, going so far as to learn how to bond the devices, running them to individual switches, learning as I went, because there was no guide for what I was trying to do.

Except, of course, there is. I just didn't find it until today:
Link Aggregation on the Dell PowerEdge 1855 Server Ethernet Switch[pdf]

Know how simple it would have been to do it right the first time if I had seen that? It's so frustrating, but the one thing that keeps me from utter despair is that it seems to be universal. I might be wrong in this, but it seems that knowledge is expensive to get but cheap to have, if you know what I mean.

Sort of reminds me of the time my boss asked me what was left to complete a project. I said that almost everything was done, but I still had one task I hadn't figured out (and I don't remember right now what it was, but it wasn't something amazingly complex) and he said, in an offhanded way, that it was trivial. I just looked at him and said "Yea, it's trivial after I figure it out. Until then, I'd say it's pretty important." It seems like a lot of things are that way.

Slow speeds transferring data between my machines

Alright, I'm not sure what I'm missing, or where I'm not testing, so I'm appealing to the people who know more than me.

I've got a couple of machines. We'll call them DB1 and DB2.

If I test the network connection between the two of them, it looks fine:

DB1 -> DB2
1024.75 Mbit/sec

DB2 -> DB1
895.13 Mbit/sec

When you convert those to Gb/s, I'm getting right around the theoretical max for the network.
(this was tested by using ttcp, by the way). So at least my cables aren't broken.

Now, the next thing I thought of was that my disks were slow.

DB1 has an internal array. It's fast enough:


[root@DB1 ~]# dd if=/dev/zero of=/db/testout bs=1024 count=10000000
10000000+0 records in
10000000+0 records out
10240000000 bytes (10 GB) copied, 58.2554 seconds, 176 MB/s

DB2 is connected to the SAN, and is no slouch either:

[root@DB2 ~]# dd if=/dev/zero of=/db/testout bs=1024 count=10000000
10000000+0 records in
10000000+0 records out
10240000000 bytes (10 GB) copied, 76.7791 seconds, 133 MB/s

When I read from the big array on DB1 to the mirrored disks, I get very fast speeds. Because my free space is small enough (< 4GB free) on my mirrored disks, I can't get a big enough file to make the transfer count. It reports 1,489.63Mb/s, which is baloney, but it lets me know that it's fast.

Reading from the SAN to DB2's local disks is, if not fast, passable
10240000000 bytes (10 GB) copied, 169.405 seconds, 60.4 MB/s
That works out 483.2Mb/s

Now, when I try to rsync from DB2 to DB1, I have issues. Big issues.

I tried to rsync across a 10GB file. Here were the results:

sent 10241250100 bytes received 42 bytes 10935664.86 bytes/sec
(10.93 MB/s or 87.49Mb/s)

Less than 100Mb/s.

I was alerted to this problem earlier, when it took all damned day to transfer my 330GB database image. Here's the output from that ordeal:

sent 68605073418 bytes received 3998 bytes 2616367.39 bytes/se
(2.62 MB/s or 20.93Mb/s)

It only says 68GB because I used the -z flag on the rsync.

To prove that it isn't some sort of bizarre combination of the SAN causing some problem when being read from rsync, here's a mirrored-disk transfer from DB2 to the root partition on DB1:

sent 1024125084 bytes received 42 bytes 11192624.33 bytes/sec
(11.12MB/s or 89.54Mb/s)

I'm willing to say that maybe the network was congested earlier, or maybe the SAN was under stress, but on an otherwise unused network, I should be getting a lot damned more than 89Mb/s between two servers on the same Gb LAN.

Any ideas?


[UPDATE]
Figured it out. Stupid compression flag on rsync

Funky Command Line

I'll take 'Command Lines that should have been shell scripts for $100, Alex'

And the answer is:

"How do I see which users have logged into my FTP server this year, how many times, and when was their last login?"

And the question:


for FILE in `tail -n 12092 xferlog | awk '{print $14}' | sort | uniq -c | sort -g | awk '{print $2}' ` ; do echo -n "$FILE: " ; echo -ne "\t\t`grep $FILE xferlog | wc -l` times. \tLast login: " ; grep $FILE xferlog | tail -n 1 | awk '{print $1 " " $2 " " $3}' ; done


Of course, this supposes that you have neglected to add xferlog to your logrotate config, AND you have exactly as many lines in your file as me. YMMV ;-)

Sysadmin Multitasking

When you say multitasking, lots of people think doing one thing while you're doing something else. For some people, the image of someone on the phone while driving comes to mind. Throw in a burger and some call waiting and you're getting to the point that most sysadmins operate at every day.

I'm going out on my own here by saying that I don't look at multitasking as a bad thing intrinsically . Other people have different opinions, but I think that multitasking really does allow me to get more done, as needed.

Pushing my judgment in this direction is the fact that I know my requirements are not "Do this, do this, do this, do this, and do this, and have them all done by the end of the day". It's "I need this, this, this and this by lunch, and this, that this that and that done by the end of the day". If I were to stop multitasking, the number of things I get done every day would half, at least. Making things easier on me is that not all of my tasks need direct supervision. Some scripts I run need checking up on to see if they're done, but if they haven't finished yet, I don't need to mess with them. This makes it possible for me to sync testing and production, replay database transaction logs, have an IM conversation with someone in another office, and write documentation, all at the same time. Doing them individually is completely inefficient.

On the other hand, some things don't compliment each other. Writing scripts and replying to email, for instance. If you're writing anything other than a completely incidental script, being drawn out of it for email means you've got to re-find your train of thought. You lose lots of time. The same goes for any in-depth activity, like configuring a lot of networking equipment from the CLI, or running an detailed query.

In the end, though, it's really up to you to decide how much (and what kind) of multitasking you can take. And regardless of what anyone tells you, it can be learned. When I was in tech support, I quickly became able to IM, email, talk on the phone, and communicate with supervisors all at the same time, and I certainly wasn't the only one, by a long shot. It comes with practice.

Trade Magazines

I've always been a sucker for getting mail. Even when I was a kid, I always used to like to have penpals, and when I started collecting baseball cards, if I got doubles of a card, I'd write a letter to that player, include the card, and ask him to sign it and send it back. It was always exciting to see who would write back and who wouldn't. Bobby Bonilla of the Pittsburgh Pirates was really nice and sent me all kinds of pictures and a personalized letter. Pete Rose sent a form letter asking for $25.

Anyway, even now, I like getting mail, which might be why I subscribe to all sorts of trade magazines and the like. I get Lightwave Magazine, from when I was going to run a mile-long fiber cable between two buildings, I get Dr. Dobb's Journal, since I used to program a bit. Of course, I subscribe to Network World, Storage Magazine, and I get all kinds of catalogs, as well.

Electronically, I subscribe to several news releases from the aforementioned sources, and of course I signed up recently with Simple Talk Exchange, as you might have heard.

Amazingly, these are all free subscriptions for "qualified professionals", which really just means you fit the description of someone who might be interested in buying things from their advertisers, Heck, I sort of like the ads, anyway. It lets me know about things that are coming out that I'll never be able to afford. Was I the only one who used to love to get Computer Shopper back when it was three inches thick?

Do you subscribe to magazines as well? Do you know of any that I'm missing? I'd love to hear about them, and I'm sure other people would as well. Thanks!

Thank you Jim!

I didn't get a chance to mention it this morning, but I really want to thank Jim for posting the entry Friday.

Jim is a relatively new admin, but he's eager to learn, which is the hallmark of a great administrator. He works in a graphic design shop and administers a pretty heterogeneous network. It'll be exciting to see where it goes.

I'm hoping that Jim will become a regular blogger here at Standalone-Sysadmin. He's got a good narrative writing style, and it's enjoyable to see his view of the system.

If you would be interested in doing a guest blog entry here, I'd love to hear your idea. Email me at standalone.sysadmin@gmail.com with the premise, and I'll let you know if I think it will fly. I'm in a stage where I can't blog as much as I'd like, and I'd love to get your writing on the site.

Shopping for email hosting

I mentioned in a previous post that I was shopping for project management software, and you guys really helped me out with a lot of great suggestions, so I'm going to try it again with another topic.

I'm looking for a hosted exchange server. I currently run CommuniGate Pro, and have for several years. I like it, I know the software, and it doesn't throw me many curveballs. However, we're moving towards a more Microsoft-centric user experience, for better or worse, and one of the things that is part of that is our mail server.

Since I don't have any Exchange experience at all, and my docket is pretty full as it is, we're looking for a hosted solution. A Blackberry Enterprise server is pretty much required as well, so is compliance (SEC/Sarbanes Oxley/etc).

One thing I've noticed while shopping is how low the guaranteed uptime is. The highest guaranteed uptime I've seen advertised is 99.9. That's ridiculously low according to enterprise standards. I suppose almost 9 hours per year isn't a long time in real-world time, but advertising only 3 9's makes me question what's powering their hosting server.

I do understand that uptime is relative, to a certain extent, but I don't want a hosting provider who just has a VPS at some other hosting company.

Do you guys have any suggestions? The companies I'm looking at right now are Intermedia, 123together.com, and MailStreet.

Interestingly, 123together has dedicated exchange hosting which they insist has a 100% uptime guarantee. I've talked to a salesman, who repeats this like he believes it, but I haven't yet talked to an engineer who can give me the actual estimate.

Of course, a dedicated hosted Exchange server is a wee bit more expensive than the shared hosting.


So, what do you all think? Do you have experience with hosted mail providers? I'd love to hear about it. Thanks a lot!

Netscreen and RADIUS

For those who have been following the blog for awhile, I'm Jim the junior Windows Sysadmin from the guest week and previously mention in other posts. Matt, has so very graciously allowed me to post blog updates so you can look forward to my experiences as I stumble my way into becoming a real sysadmin.

As noted in my previous blog post my current project has been a new PDC. Unfortunately, due to issues with third party apps and 64bit Windows, I have had to nuke the install and start over. Beyond that, implementing all the services that I had running on our old 2003 box on the new 2008 box has been almost too easy. I even got a new certificate server running on it without hitting a snag. The only thing that has really stumped me was the RADIUS configuration.

To give a little background, We have two RADIUS clients; a Netscreen firewall and a Cisco wireless access point. The guy whose job I inherited initially set up RADIUS and configured the Netscreen client and applicable policies, while my boss configured the Cisco and it's policy. However, the initial notes (if there were any) have been lost in the sands of time, so I have been flying sort of blind here. And of course, Microsoft went and redid the old Internet Authentication Service (IAS) and turned it into the Network Policy Server(NPS) on W2k8. And while NPS is actually pretty easy to set up, things are located in different places and it makes it more difficult to do a side by side comparison to set things up.

Setting up the Cisco was pretty trivial, it uses no fancy settings, I just had to tell NPS that it was authenticating 802.11 wireless clients and it was good to go. The Netscreen however caused some problems. Since I had no idea how it was originally set up I had to dig through the settings to figure out how the authentication was supposed to work. I found that there are two external groups on the netscreen that correspond to two groups within the Active Directory: Domain Admins and WebAuth. So the first thing I tried was setting up a network policy for each that had a condition where the user had to belong to the group in question. And of course, it didn't work. So off to the Juniper Knowledge Base I go. This led me to these pages here and here. And that gave me the final clue (and I only felt a little foolish for missing the setting on the old DC). I did not know I needed to specify an Vendor Specific Attribute to make it work. I set up Attribute 3 (specify an external group) and I was off and running.

And yes, I did document the process this time.

Thankfully the only things left on this project are setting up DHCP (trivial is the service will remain running) and raising the domain functional level, but I will leave that for the weekend I think.

Great News!

I'm very happy to say that this morning, I completed discussions with Red Gate Software to become a regular columnist in their magazine, Simple Talk Exchange!

Simple Talk is a journal for IT administrators, and deals primarily with Microsoft technologies such as SQL Server, .NET, and Exchange. I don't run any of those technologies myself, but my column will focus more on what it's like to be an administrator, and deal with the issues we face, rather than any particular technology.

The column will be entitled 'From the front line - the confessions of a SysAdmin'. If you're interested, you should sign up to receive the magazine. It's free, and a good source of information. Look for my articles to start after the October issue.

And just so you know, this really won't affect my blogging, except I'll probably link to my articles when they come out.

It's the little things that get to you

Lots of low to mid level networking hardware gets configured with web based interfaces. This is typical, whether it's an ap, a router, switch, firewall, or what have you. CLIs seem to be going the way of the dodo, at least for low-cost equipment.

One of the consequences of this is a deplorable lack of quality of the web interfaces. I've seen everything from help buttons that didn't react at all, to using an iframe to load documents from the company's webpage, to all sorts of illogical inconsistencies across the interface.

One of my pet peeves has always been IP configuration. During the initial configuration of the equipment, you've invariably got to change the IP address. You're doing this with a web based tool. You know that, once the IP address changes, you're not going to be able to get to the device on the old IP. Hyperlinks were invented in 1965. Why, in 200X, do so many manufacturers just make a link that says "Continue" and points them to the config root on the same IP? Doesn't that seem counter-productive?

Finally, I've encountered someone who said, "You know, maybe we could link to the IP address that this thing was changed to...". My new 3com Baseline 48port switch has this feature. I'm irrationally impressed by this small detail.

Like the title says, it's the little things sometimes...

Ohio LinuxFest 2008

I recently received an email from the Ohio Linuxfest organizers reminding me to register.

Historically, I haven't been able to attend, as it was the same weekend as my anniversary, but this year, it's October 10th and 11th, so I'll be able to attend, provided I've not relocated to NJ by that point.

It looks like it'll be exciting! If you pony up the big bucks ($350) for a Professional package, you get to attend the OFLU, with classes being taught by several people that you might have heard of. :-)

Anyway, I'm hoping that I can make it, and if you're in the area, you should too.

Recovery and return to normalcy

With the return of electricity to my home last night, I found a huge wave of stress leave me. Aside from having to use lanterns and candles for light, not being able to use hot water for cleaning, and spending all day trying to work from wherever I could find internet, it wasn't too hard to live without electricity, but as a sysadmin, the knowledge that I might be forced to leave and try to find an open AP at 4 in the morning because of a system outage weighed very heavily on me.

I really felt like a technomad of sorts. Breakfast, lunch, and sometimes dinner had to be held at places with open wifi. Of course, just finding a place that had power in the beginning was a little iffy.

Yesterday I worked from the office, and I was surprised at how much more relaxing that was, than even sitting in a pub for lunch. Almost all the work that I do is remote to NJ, but somehow just the knowledge that my resources were around me helped. The dual widescreen monitors might have had something to do with it as well.

At least it is now over. I'm hoping to begin blogging more often again, too. It was surprising how much I missed it. It really is cathartic to put everything out here and get input from people who have to go through the same things as myself. Thanks, everyone for all of your support! I do appreciate it.

Post-WindStorm Update

I realize that I've been too busy to post recently, but man, I'll take normal busy over the past two days anytime.





For those of you who don't know, I live in central Ohio, in Columbus. The other day, the remnants of Hurricane Ike blew through. We received 75mph wind gusts, with around 50mph sustained. The net result is that neither my house nor my office has had power since Sunday afternoon.

I'm typing this from Panera, where I've been working this week. Free wifi and crusty bread make for a powerful draw, but the vanilla classical music wears on you. We got word of a local pub with wifi, so we're going there for lunch :-)

I don't know when I'll be getting power, but here's the central Ohio power outlook for the moment:



We've got 48% of the people in Franklin County that don't have power. It's exciting.

Anyway, trying to run things from Panera is a little rough. Last night, it was at capacity and kicking people off. I don't imagine things are better by much today.

If you want to see more storm damage, click the tree above for the Flickr photo set.

[UPDATE] It can always be worse: http://it.toolbox.com/blogs/db2zos/hurricane-ike-update-27201

Behind Schedule

I've been posting less and less frequently over the past week. It's not because I don't want to, it's just because work has been so crushing, in terms of needing to get things done.

On deck is a writeup on configuring and managing log files, and a big howto on clustering using RedHat Clustering Suite, which you can actually use for free via CentOS.

We're in our last big push towards getting the system usable for operations testing. This includes such niceties as getting the DNS configured correctly using the proper naming prefix. Despite my making clear what form the DNS names would take, I have been over-ruled. Instead of a-<function><servernumber>, we're doing "<function><servernumber>" for the primary, and "<function><servernumber><postfix>" for everything else, where postfix is back, test, etc. So what was a-fs1, for "first fileserver at the alpha site", it's now fs1. b-ops3 for "3rd operations machine at the beta site" is now fs3back. I don't like it, but it's not my choice.

I'm working on configuring Sendmail now, and I'm surprised someone managed to transcribe the documentation from the original cuneiform. I've never really worked with Sendmail, as postfix always seemed much easier to deal with. I'm still weighing whether it's better to go with the default install on the OS (sendmail) or require a custom install for the machine. I've got to admit that I'm leaning towards postfix.

Anyway, that's what's going on with why my updates have been so infrequent.

This has been the most blog-like entry I've done so far, I think. All I need to do now is go all emo and write poetry.

Followup of Followup of Downtime

Bob, over at lonesysadmin.net is also covering downtime today. If you're interested, you should definitely check it out.

Followup on downtime computations

last week, I posted a quick blurb about calculating minutes of downtime per 9. In one of the comments, Jeff Hengesbach wondered about how other people calculate uptime.

The particular question was whether you, when computing your availability numbers, should include planned downtime in the final numbers. In other words, if your maintenance window is from 1am till 2am, do you count that hour-long window as downtime.

I can see both sides of the argument, but my general mind is that as long as you are up front about your maintenance windows, and the customers have no reasonable expectation of service during those published times, then it shouldn't count off your downtime.

However, sometimes service availability is of paramount importance. In these cases, you typically don't have any downtime, as normally recognized. In my case, I provide financial data to timezones spread all over the world, which means that there is literally no time that someone isn't trying to access services. I literally have no downtime from a customer's perspective.

To combat this, I use an array of methods to provide redundancy at every level. Though my budget doesn't allow me to use truly enterprise level solutions, I try to do the best with what I have and what I know. Every iteration of the infrastructure has improved upon the previous one. I'm currently using technologies like blade servers, a small SAN, and a cluster of linux machines, along with redundant switches and network access. In addition, we never have less than two physical sites. For me, any time a client can't access any part of the service counts off of our availability.

I am curious too, how everyone else does it. What do you count against your uptime. Do you even keep track of your uptime or availability? I've never had anyone ask me what mine is, and to be completely honest, I have never actually tracked it. It's always just been "down as little as possible", but I'm rapidly approaching the time where the Mean Time Between Failure (MTBF) of the equipment counts, and I may be asked what my uptime has been and is calculated to be. I don't want to have to make stuff up, so I'm appealing once again to everyone else.

How do you do it?

Downtime Computations

I'll probably write a much longer article on this later, but in case you're curious about "Four 9's of uptime" is, just google it:


http://www.google.com/search?hl=en&q=one+year+minus+99.99+percent+of+one+year&btnG=Search

Telephone conference calls are a lot like ethernet

Ever notice when a broadcast question is asked on a conference call, people respond pretty much automatically with CSMA/CD? Just a thought I had the other day.

Random thought on iSCSI vs FC

I read Jeff Hengesbach's blog every day, and I'm fairly sure that if you like my blog, you'll love his. He recently started it, and every post so far has been really interesting. You should check it out.

Anyway, today Jeff wrote about iSCSI SANs. If you're unfamiliar with the concept, here is a great introduction. You should also know that there are two competing standards, iSCSI (where the storage is accessed over standard ethernet) and Fibre Channel (where the storage is accessed over fiber optics cables). There's some more in-depth explanation here, but it all boils down to speed and expense.

Among the in-the-know people (AKA: not me), it seems to be a sort of common knowledge that we're going to be using iSCSI in the future, what with 10Gb ethernet being the standard just-around-the-corner. The current best-of-breed is 8Gb/s FibreChannel from IBM, but 10Gb would trump that, and the added bonus of using tried and true ethernet is very appealing to a lot of people.

Regardless of whether or not we end up with iSCSI I'm not convinced that copper will last us much longer in the grand scheme of things. I suspect that, maybe by the time 100Gb rolls around, we'll be using ethernet-over-fiber for that stuff. The theoretical bandwidth of optical is just too high to ignore in the long-term. If you want to get all sci-fi, scientists have recently began working on entirely optical versions of most basic integrated circuits.


Just a random observation to spark discussion. What do you see happening in the future of SANs?