Configuring the Netgear SSL VPN Concentrators (SSL312/FVS336G) with Active Directory / LDAP

If you're a chronic Standalone Sysadmin reader, you might remember that a while back, I started implementing an Active Directory infrastructure with-which to centralize authentication for my Linux hosts. Well, getting the Linux machines authenticated has been completed, and now I'm on to further integration.

I've talked about my VPN issues before, and that I picked up some SSL VPN concentrators. An added feature of the ones I got are the ability to authenticate against Active Directory and/or LDAP. I figured it was time for me to put it to use.

Now, I'm no old hand at Active Directory...I've got the kind of knowledge that comes from reading several books but never touching it; in other words, the kind that leads to pain and gnashing of teeth. When I started creating my AD users, I just had employees, so I created users in the "Users" folder. Straightforward enough. We've got several classifications of users, and many users are in multiple groups, which in Windows is easy enough. I used a pretty much 1-to-1 Unix group to Windows security group (not Windows distribution groups, which is only for email (thank you SAM's ActiveDirectory Unleashed)). The group assignment was simple.

Then I thought about it, and remembered that we really have a couple of different bodies of users. For example, some of our clients have FTP accounts that they connect to in order to drop off or pick up files. There wasn't any sort of hierarchy under "Users", so I created two new security groups: the first, employees, which contains all of my employees, and clients, which contains all of our clients. I restricted the accounts in AD so that the clients could only login to the FTP servers, and I setup Likewise-Open so that only accounts in the "clients" group could FTP in, and only accounts in the "employees" group could connect to the rest of the machines. Theoretically, all of the other machines were inside the network, and behind the firewall so there's no chance a client would be logging in anyway, but there's no sense not being thorough.

All was well and good until I went to set up these VPN boxes. The only fields I could fill out were "authentication server", which was the local domain controller, and domain. Well, both answers were straightforward enough, except that if I set it up that way, all of the "client" users would be able to log in and get a VPN connection to the inside network. I tested it, and was right. Not a good thing.

I read a few help files and some documents on the devices, and found a suggestion for limiting group access. It suggested pointing to the OU (which is sort of like a folder in LDAP terms) that contained the appropriate users via LDAP authentication, rather than a direct "Active Directory" connection. Erm. Okay?

In an aside, I knew that Active Directory was essentially a gussied up LDAP server, but I didn't (and don't) know all that much about LDAP. I have a really big LDAP book that I've skimmed part of, but to say I have a mastery of it would be laughable. I know that "DN" is distinguished name, and "OU" is "organizational unit", and there's some sort of hierarchy for when you are building DNs. Or something like that.

So I read, and researched, and played, and installed the ldap tools package, and researched some more. And made liberal use of the "ldapsearch" command, and found this post which taught me how to query the Active Directory server from the command line. And it was good.

I read some more, and played some more, and came to the sad realization that I couldn't make my VPN boxes authenticate against the AD LDAP unless I modified it to create an OU to hold the accounts that I wanted to allow access to.

When you're faced with a problem that you know little to nothing about, and you want to test an idea that you suspect might work, but might also break the entire infrastructure you've spent the last few months of your life building, it's a good idea to get a second opinion.

That's why I gave my good friend Ryan a call (he's on hiatus from his blog while he assembles a datacenter from a box of erector sets), who knows far more about AD than I do, and explained the situation. I said that the manual suggested pointing to an OU, and that my research suggested that I might want to create another OU for the accounts to live in, but I was concerned that there was some sort of "Windows Magic" that would be broken if I just started to move accounts to this new "OU" all willy-nilly.

Ryan suggested making two OUs, one for "internal" accounts, and one for "external" accounts. Then, and when he said this, I smacked myself in the forehead, he suggested making "test" accounts in the Users folder, verifying that they worked, and then moving them, and seeing if they still worked. Ryan is a brilliant guy, and I owe him a few more beers now :-)

So I followed his suggestions, created the OUs, created the test user, it worked fine, tested transitioning my account, and it worked fine, and then I tested moving the client FTP accounts. They worked fine. I had created the OUs, moved accounts, and nothing broke. Glorious.

Time to get the VPN machines to authenticate. I created a new domain, using LDAP authentication, and it asked for the server address and the base DN. The server address was just the IP, and I had gotten good enough to know that the base DN was going to be "OU=Internal,DC=mydomain,DC=TLD". I saved it, opened another window and tried to log in with my domain credentials. And failed.

I thought about it, and remembered from doing the command line LDAP queries that my Distinguished Name (DN) actually started with "CN=Matt Simmons" rather than msimmons@mydomain.tld. On a hunch, I tried logging in with a username of "Matt Simmons" (without the quotes) and my domain password. Light shone from the heavens, choirs of angels sang, and I got the VPN portal.

That, my friends, was my experience Thursday. I've learned a lot, and I feel a lot more confident about LDAP and Active Directory. And I'm able to continue to centralize user administration. It feels pretty good.

I'm really interested in what other people are doing with Windows servers and Active Directory. Are there tips that you've picked up on the job and want to share? I'm really open to suggestions on what I've been working on too. I know so little that almost everything I hear is new information. It's an exciting phase for me.

Default vsftpd on CentOS is dumb

This is pretty ironic, since the vs stands for "very secure".

From the top of /etc/vsftpd/vsftpd.conf:

#
# Allow anonymous FTP? (Beware - allowed by default if you comment this out).
anonymous_enable=YES
#

This is the default configuration. Now, one of a couple things is going on here. Either the comment is lying, or the configuration flag is lying, or I'm terribly confused about what these words mean.

I figured that I'd check to see which was the case:

msimmons@newcastle:~$ ftp ftp1.test
Connected to ftp1.test.
220 (vsFTPd 2.0.5)
Name (ftp1.int.ia:msimmons): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

OKAY! This isn't good. In fact, it's a Bad Thing(tm).

Lets fix that. Ignoring the utterly stupid comment, I switch the flag to "NO", and restart the daemon. I try again, and I fail. Horray. Lets see what else I can find.

I log in as a pretend user, and I authenticate fine. I 'cd' to .. and do a directory listing, and what do I find, but all of the various client accounts. Our clients are confidential, which means that them seeing each other would be a Bad Thing(tm). I dig into the config again, and find this gem:


# You may specify an explicit list of local users to chroot() to their home
# directory. If chroot_local_user is YES, then this list becomes a list of
# users to NOT chroot().
#chroot_list_enable=YES
# (default follows)
#chroot_list_file=/etc/vsftpd/chroot_list


Great, so apparently, I just need to find and flip the flag on chroot_local_user. Of course, it doesn't exist in the file. I create it, set it to "YES", restart the daemon, and things are working the way they should.

The question in my mind is why an FTP daemon that bills itself as Very Secure comes with such an asinine configuration. There are occasions where chrooting the ftp users isn't called for, but there are relatively few occasions that require anonymous FTP access. I can't understand why they wouldn't have shipped a secure config and then made people unsecure it, as opposed to the way it is now. Really hard to believe.

I suppose it is possible that the distro maintainers are responsible, but it's still stupid.

Unix Rosetta Stone

I just found the Unix Rosetta Stone which seems simplified, but still probably handy if you've got a really heterogeneous network, or if an AIX machine should suddenly spring up in the middle of the server room.

Judging from the number of Delicious bookmarks it has, it's pretty well know, but I figured that I couldn't be the only person in the dark, and I figured someone might get some info from it.

Wacky SSH Authorized Keys Tricks

You may have caught my blog post last week about setting up host to host ssh keys.

What you might not have caught was in the comments, where Ben Cotton mentioned a trick I hadn't heard of, namely specifying the allowed remote commands in the authorized_keys line. He said there were even more features available, just waiting on the manpage. I replied that if he wrote it, I'd link to it.

Well, Ben put his money where his mouth is. He goes into nice detail and provides some good links and suggestions. This is really fascinating stuff, and I'm looking forward to using it in my own organization.

Therek over at Unix Sysadmin jumped in the fray, too. He's got three neat tricks for your ssh needs that you should really check out. I had no idea SSH key auth could be bent in these directions!

I've said it before, but I'll keep saying it. I love having visitors to my blog who enjoy what I write, and it really brings it home to interact with everyone like this. I couldn't ask for a better bunch of readers, though to be honest, I'm worried about Ben's longevity. I can't imagine what his cholesterol level must be ;-)

Ben, Therek, thank you both very much! I know my readers will really enjoy these articles. And as for everyone else, the same offer goes for you. If you've got something to share, let me know, I'll be happy to link to your blog entry or host it here if you've got the urge to write.

Great tool for network diagramming

I'm getting ready to implement a new Nagios monitoring system at our soon-to-be-production server, and I'm using Nagios v3 this time. Because I sort of figured out the configuration on my own last time, and it grew in a very organic (read: unplanned) way, the config is a mess. That is going to be different this time, thanks to Nagios 3 Enterprise Monitoring. It's not an intro guide to Nagios, that's for sure. The first chapter deals with what's new. The 2nd chapter deals with streamlining the configuration for large installations. It's been very educational in teaching me how various hostgroups and service groups can work together to really make life easier for configuring monitoring.

After reading this book pretty much cover to cover, I decided that I needed to logically map out the various relationships of my services, to figure out the inheritance policies (Nagios supports multiple inheritance in configuration objects).

I started looking for a good free diagramming tool, first on Windows then on Linux. Windows was hopeless. I found lots that looked promising, but ended up being shareware. I don't have MS Office Pro on my personal laptop, so I didn't have Visio handy, and I wasn't going to buy a piece of software when I was sure that something good and free existed.

Giving up, I booted into Linux to see if anything I didn't know about was in synaptic. Of course not. The best diagramming solution in Linux is Dia, and I'm sorry to say it, but it's ugly. Really ugly. I'll use it if that's the only thing available and I'm just looking for something quick, but I won't like it.

I kept looking, and finally out of desperation I did a search for online applications, and I hit the jackpot. I found Gliffy. It's a flash diagramming application with built in stencils for all sorts of things, and the ability to add your own clipart. It'll even export to Visio.

I was impressed. It's free for personal use up to 5 public diagrams. You can pay $5/month for unlimited drawings and removing the ads, and there are corporate versions that have built in collaboration. It's easy to use, and it helped me a lot. Here's a drawing of some of my nagios groups:



If you're in the market for a cross-platform diagramming solution, you could do a lot worse than Gliffy.

Building and designing systems: Is the cart pulling the horse?

There's a really interesting post over on Code Monkeyism about test driven development of code, and how it's related to the design of the space shuttle engines.

The short of it is that opposed to typical complex engine designs, where each individual part was tested independently and then together in subassemblies, and then again when the unit was complete, the space shuttle was pretty much designed, assembled, then tested. The better method has the advantage of weeding out all the really bad decisions in the small scale, then when you get to the point that you put them together, it generally works rather than flying apart at high speed.

While Code Monkeyism is primarily centered on software development, the points that Stephan make are readily applicable to us as infrastructure engineers, particularly in a growth phase where we're engineering new solutions and trying to implement them.

I'm as guilty of putting the cart in front of the horse as anyone. My debacle with the cluster was a prime example. When you're given a job to do, the equipment to do it with, and no time to learn, these kinds of things happen. Particularly when you're working with shoddy tools anyway.

I shouldn't have attempted to have the very first cluster I created be a production system, first. More due diligence in researching solutions was called for, and I probably would have learned beforehand that RHCS wasn't ready for prime time. I have learned from the experience, though, so all is not lost. Using the knowledge and experience I've gained, the next time will be more solid.

Is this something that everyone has to learn on the job, or was there a class or memo that I didn't get?

Host to host security with SSH Keys

I have a lot of Linux hosts. Somewhere in the vicinity of 70 servers, all but 3 run some variant of Linux. Lots of them have to communicate seamlessly through cronjobs and monitoring daemons. To pull this off, I've implemented SSH key authentication between the applicable accounts. The method is pretty easy.

Check the ~/.ssh directory for the user you want to ssh as. There's probably a "known_hosts" file, which keeps track of the machines that user has contacted previously, and there's probably an id_dsa and id_dsa.pub. These are the private and public keys of the user, respectively. You might instead see similar files, but with "rsa" instead of "dsa". These are keys that have been created with another encryption method. See more information here.

We have the keys now, so what we want to do is make the remote machine aware of them, so that our account on the source machine which has the private key can connect without authenticating with a password. To do this, we install the public key (the id_dsa.pub) in the ~/.ssh/authorized_keys of the remote account we want to connect to, on the remote host. So, we have

Machine A:
User: msimmons

Machine B:
User: msimmons

machineA$ cat ~/.ssh/id_dsa.pub
[text output]

machineB$ vi ~/.ssh/authorized_keys
[insert text output from machineA]

Ensure that the permissions on the authorized_keys file are not world-writable or the ssh daemon will probably refuse to connect. It should also be noted that your sshd config (probably in /etc/ssh/sshd_config) should be setup to allow key based authentication. The manpage should help you there.

At this point, you should be able to connect from one account to the other without a password. This allows you to use rsync to transfer things automatically, through the cron. It would look a bit like this:

machineA$ rsync -e ssh -av /home/msimmons/myDirectory/ msimmons@machineB:/home/msimmons/myDirectory/

Read the manpage for (many) more rsync options.

There is a weakness to this method, though. Anyone that obtains a copy of the private key (the one in machineA called id_dsa) can pretend to be you, and authenticate as you to machineB (or any other machine that has your public key listed in the authorized_keys). This is potentially a very bad thing. Particularly if you have your private key on your laptop, and the laptop gets stolen. You wouldn't want a thief to get their hands on your private key and compromise the rest of your network. So how to get around not needing a password, but not wanting someone just to be able to use your private key if they get a copy. The answer is to use a pass phrase on your private key.

Through proper use of the ssh-agent and ssh-add commands, you can set up passwordless communication from one machine to another. I could explain the common usage of these, but it would just be duplicating this fine effort from Brian Hatch: SSH and ssh-agent. He talks about setting up ssh-agent and ssh-add, but if you're like me, you've already got existing SSH keys laying around without passphrases. The answer to that is to simply run ssh-keygen -f [keyfile] -p to reset it.

Now that you've got a working secure key and a way of not having to type your passphrase every time, lets figure out how to get your servers to take advantage of the same technique. At the very least, you're going to have to type the user's passphrase once, either the first time you want to connect, or (more likely) when the machine boots up. That is not to say that you'll require a password to boot the server, just that before your cron jobs run, you'll need to start the ssh-agent.

Once you start the ssh agent on the remote machine and add the key (per the instructions above), how do we keep that information static? Well, remember those variables that ssh-agent setup that tell 'ssh' the socket and PID to talk to the agent with? It turns out that you can put those (and any other variables you need to be static and universal) in the crontab at the top:

msimmons@newcastle:~$ crontab -l
SSH_AUTH_SOCK=/tmp/ssh-sBrpd11266/agent.11266
SSH_AGENT_PID=11267
48 10 * * * ssh root@testserver.mydomain uptime > ~/uptime.txt 2>&1

This will allow any of the scripts being called by the cron daemon to access the variables SSH_AUTH_SOCK and SSH_AGENT_PID, which in turn allows your scripts to ssh without using the passphrase. All that is required is updating the crontab when you reboot the machine and/or restart the agent.

On my desktop, since I ssh a lot, I add the same variables to my .profile in my home directory so that I only need to type in the passphrase once. If you find yourself connecting to other machines frequently from the server, you might want to do the same thing.

I'm sure I messed up the explanation in some parts, so if you have any questions, please don't be afraid to ask in the comments. I hope this helps someone set up their key-based authentication in a more secure manner.

[UPDATE]
See the followup to this article!

Datacenter that could belong to S.P.E.C.T.R.E.

This datacenter is dripping with "evil lair" vibes.

The Worlds Most Super-Designed Data Center

Tips for an initial buildout

St Aardvark the Carpeted (best. name. EVER.) has been working on building out a new data site for one of the companies he works for. He's got some great tips on things to remember and take into account before you do one of these yourself.

I know the installers that built my most recent rack at the colocation really appreciated the
layout that I made to show what was going where in the rack. I also prepared spreadsheets listing all the cables and where they went. The colocation also needed all serial numbers to all equipment that I was bringing in, which is good for me to have anyway and is probably a good practice to have on hand.

Has anyone else got any tips for a one-time build out that would help?

Sysadmin Extorts company for better severance package

I just read this over at TaoSecurity. Apparently a recently laid-off sysadmin was arrested because he threatened to bring down the IT infrastructure if his severance package wasn't improved.

This isn't the first disgruntled sysadmin story we've seen this year. Please, I beg of you, spare the servers in your rampage. ;-)