Host to host security with SSH Keys

I have a lot of Linux hosts. Somewhere in the vicinity of 70 servers, all but 3 run some variant of Linux. Lots of them have to communicate seamlessly through cronjobs and monitoring daemons. To pull this off, I've implemented SSH key authentication between the applicable accounts. The method is pretty easy.

Check the ~/.ssh directory for the user you want to ssh as. There's probably a "known_hosts" file, which keeps track of the machines that user has contacted previously, and there's probably an id_dsa and id_dsa.pub. These are the private and public keys of the user, respectively. You might instead see similar files, but with "rsa" instead of "dsa". These are keys that have been created with another encryption method. See more information here.

We have the keys now, so what we want to do is make the remote machine aware of them, so that our account on the source machine which has the private key can connect without authenticating with a password. To do this, we install the public key (the id_dsa.pub) in the ~/.ssh/authorized_keys of the remote account we want to connect to, on the remote host. So, we have

Machine A:
User: msimmons

Machine B:
User: msimmons

machineA$ cat ~/.ssh/id_dsa.pub
[text output]

machineB$ vi ~/.ssh/authorized_keys
[insert text output from machineA]

Ensure that the permissions on the authorized_keys file are not world-writable or the ssh daemon will probably refuse to connect. It should also be noted that your sshd config (probably in /etc/ssh/sshd_config) should be setup to allow key based authentication. The manpage should help you there.

At this point, you should be able to connect from one account to the other without a password. This allows you to use rsync to transfer things automatically, through the cron. It would look a bit like this:

machineA$ rsync -e ssh -av /home/msimmons/myDirectory/ msimmons@machineB:/home/msimmons/myDirectory/

Read the manpage for (many) more rsync options.

There is a weakness to this method, though. Anyone that obtains a copy of the private key (the one in machineA called id_dsa) can pretend to be you, and authenticate as you to machineB (or any other machine that has your public key listed in the authorized_keys). This is potentially a very bad thing. Particularly if you have your private key on your laptop, and the laptop gets stolen. You wouldn't want a thief to get their hands on your private key and compromise the rest of your network. So how to get around not needing a password, but not wanting someone just to be able to use your private key if they get a copy. The answer is to use a pass phrase on your private key.

Through proper use of the ssh-agent and ssh-add commands, you can set up passwordless communication from one machine to another. I could explain the common usage of these, but it would just be duplicating this fine effort from Brian Hatch: SSH and ssh-agent. He talks about setting up ssh-agent and ssh-add, but if you're like me, you've already got existing SSH keys laying around without passphrases. The answer to that is to simply run ssh-keygen -f [keyfile] -p to reset it.

Now that you've got a working secure key and a way of not having to type your passphrase every time, lets figure out how to get your servers to take advantage of the same technique. At the very least, you're going to have to type the user's passphrase once, either the first time you want to connect, or (more likely) when the machine boots up. That is not to say that you'll require a password to boot the server, just that before your cron jobs run, you'll need to start the ssh-agent.

Once you start the ssh agent on the remote machine and add the key (per the instructions above), how do we keep that information static? Well, remember those variables that ssh-agent setup that tell 'ssh' the socket and PID to talk to the agent with? It turns out that you can put those (and any other variables you need to be static and universal) in the crontab at the top:

msimmons@newcastle:~$ crontab -l
SSH_AUTH_SOCK=/tmp/ssh-sBrpd11266/agent.11266
SSH_AGENT_PID=11267
48 10 * * * ssh root@testserver.mydomain uptime > ~/uptime.txt 2>&1

This will allow any of the scripts being called by the cron daemon to access the variables SSH_AUTH_SOCK and SSH_AGENT_PID, which in turn allows your scripts to ssh without using the passphrase. All that is required is updating the crontab when you reboot the machine and/or restart the agent.

On my desktop, since I ssh a lot, I add the same variables to my .profile in my home directory so that I only need to type in the passphrase once. If you find yourself connecting to other machines frequently from the server, you might want to do the same thing.

I'm sure I messed up the explanation in some parts, so if you have any questions, please don't be afraid to ask in the comments. I hope this helps someone set up their key-based authentication in a more secure manner.

[UPDATE]
See the followup to this article!