Back to a normal week (hopefully)

I'm back in Ohio this week, and it's nice to have things return to a relative normal. On the personal front, we lowered the price of our house to get it to sell faster, and we've got some apartments in NJ that we're pretty sure we'd like to move into. We've just got to find someone who wants to buy our house now.

At work, my boss is driving up to NJ today, moving stuff from the office, so when I get there, I know it's going to look even more empty than it did.

On the technology side, we finally had to pull the plug on Redhat Cluster Suite. I don't think I've ever invested more of myself in trying to make something work, and still have it fail. I plan on publishing all of the work and configs that I did in the hopes that it helps someone else, but at this point, it's not stable enough for us to use in production. We kept getting the cluster hung on certain filesystems and even when it didn't hang, performance was unbelievably slow, even for doing an 'ls'. Anything having to do with metadata of more than a few dozen files froze for a long time, and trying to run 'find' on a clustered directory was enough to kill access to that volume. I suppose for something smaller than ~2million files, it might work. Next time I think we'll just get a NetApp. When you consider the amount of time I've spent working on this, it'll be cheaper.

To tell you the truth, I'm just shellshocked. I really did spend months on it, and I let my emotion get in the way of my judgement. If I had been looking at it from a detached viewpoint, I would have pulled the plug a long time ago, but I wanted it so bad that I kept hoping it would stabalize.

For an alternate solution, we're running active/passive server backups, where the backup server is alive, but doesn't have access to the volumes. I've had to do all kinds of key magic to make things work right with ssh, I'm going to have to write scripts to switch smoothly to the backup server. I'm sure there are other ramifications that will occur to me, too.

All in all, I'm really glad that last week is over.