Doing it securely with OpenSSH

Today, in the Unix world, remote administration is a must. Telnet, while still used here and there, is awfully insecure, old and useless for the task. Most, if not all, administrative tasks can be performed on the command line, so the only universal choice left is OpenSSH/SSH...

Disclaimer

Here I'm sharing my knowledge about OpenSSH and other tools. This article is not an OpenSSH security reference or something. For detailed description of the options consult the manual pages. If you find any mistakes or have more interesting tips to share - feel free to contact me or comment below. And last, but not least, SSH is a vital part of your security infrastructure, so before trusting me with your servers please double check everything. If at some point you start trusting me too much, double check again - you must know you still can. :-) A little bit of paranoia is always a good thing(tm) and (almost) never hurts.

Tools for remote administration

If you are already in love with OpenSSH you can skip this and the next section. There are many ways you can remotely access and administer your machines (I'm not talking about managing file and web sites). The solutions range from home-built ones, telnet over VPN connections to Internet standards like SSH. There are graphical applications (meaning you see your graphical user interface from the remote machine) like VNC, but they are no go here. Running graphical application on the remote end to show you a simple console is much much slower and in short - out of the question. In the Unix world the console is a must - having no console access is like loving a girl without ever touching her... ;-) In addition VNC (for example) must be run over SSH tunnel in order to get reasonable security. Because I can't possibly discuss the home-built solutions and all graphics only ones are out of the way (remember the girl?), let's see what's left.

Telnet. The first thing I want to shout at you about telnet is a big fat warning: DO NOT USE TELNET! Well, at least do not use it on unencrypted, insecure or public networks without a VPN tunneling it's traffic! Every recent packet sniffer can catch telnet user names and passwords on the fly in the wink of an eye! Telnet did it's job long ago, and it did it well, but now it's just too old and insecure. Use telnet if nothing, I really mean it, nothing else is available! There are still some devices that only support telnet, but these are slowly going into the computer history...

So, the only good choice left is SSH. It's secure, it's fast, it provides a number of authentication methods, it's Internet standard and it offers much more than simple command line interface. What else you ask? Well... some of the *other* things SSH offers are:

running commands on the remote host (obviously)
a number of authentication methods - password (username + password), publickey (key based authentication with RSA/DSA and other keys), X.509 certificates, keyboard-interactive (like password, but can use PAM, One Time Passwords, SecureID and others), GSSAPI (provides external authentication like Kerberos 5, NTLM and other).
redirecting the input/output of remote to local programs and the reverse (telnet still can't).
X11 forwarding (working with the GUI of graphical programs, which are running on the remote host).
forward/reverse port forwarding - you can connect to tcp service on the remote LAN, which is not available from outside the LAN, just connect to a machine there.
sending/receiving files - 'scp' and 'sftp', these two can do ftp's job just fine in the most cases.
SOCKS5 proxy - you can tell your browser (FireFox/Mozilla/Netscape at least) to use it and the world will see you like you are on the remote machine.
tun network interfaces - the most recent versions of OpenSSH can create VPN connections over SSH's encrypted channel. There's also ongoing effort to make OpenVPN and OpenSSH interoperable - creating encrypted tun interface connecting with ssh client to OpenVPN server and the reverse.

I'll address some of these features in the following articles, now we'll focus on securing and configuring OpenSSH, sshd mostly.

Interoperability

Before jumping in, let's see how good choice SSH is in interoperability. You can connect to your server from virtually everywhere - every Unix like system, Windows machines, Macintosh machines, even from your mobile phone (I'm not saying the experience is going to be a very pleasant one). For the small mobile devices there are java SSH client implementations, so most modern phones will do the trick. It can be painful to type commands from your phone, but it can be a life saver (ask me about fixing my core router from a taxi in 06:30 in the morning after a lo-o-ong night if you ever meet me). In short - it is all one can ask for. Well, it won't make you a coffee at 02:00 AM, but... let's jump in anyways!

Running OpenSSH

There are different SSH implementations - commercial: SSH by SSH Communications Security (the original), F-Secure SSH, and the FLOSS alternatives: OpenSSH (forked by OpenBSD from the open-sourced SSH-1 version, now ported to many platforms, the one we'll discuss), PyTTY (excellent windows client), LSH (for embedded devices) and more. You can take a look at http://www.snailbook.com/software.html for more information (capabilities, OS-es supported, SSH protocol versions...). Which version you will use is up to you in the end, but here I will stop your attention on OpenSSH. Why OpenSSH? Because... it comes with every recent Linux distribution (there are rumors some people have seen distributions without, buy I'm told they have seen dinosaurs also :->) and because it is very powerful. The first thing you need to do (if not done by the installer for you) install it, configure, test, love, make it run every time your machine starts and use it every possible second :-). The installation on Fedora/RHEL/CentSO can be done with 'yum install openssh-server', for other distributions please consult your documentation.

Running SSH a inetd service

You can run OpenSSH is as an inetd service. An example configuration for xinetd is:

service ssh-backup
{
    disable		= no
    socket_type		= stream
    protocol		= tcp
    port		= 2022
    type		= UNLISTED
    wait		= no
    user		= root
    server		= /usr/sbin/sshd
    server_args		= -f /etc/ssh/sshd_backup_config -i -b 1024
    log_on_failure	+= USERID
#   only_from		= 127.0.0.1 10.15.0.0/24
}

Save this file as in /etc/xinet.d/ssh-backup (thanks for the nice sample go to Field Commander Wieers, http://dag.wieers.com/, hmm... I see my name there, thanks Dag!) copy the configuration

cp --preserve=all /etc/ssh/sshd_config /etc/ssh/sshd_backup_config

... and restart xinetd ('service xinetd restart' on RedHat like systems). This will run ssh on port 2022 (the standard is 22) with alternative configuration file. This same thing can be done with the old inetd daemon, but it's not as powerful and nice.

This method has the only two advantage - you have no OpenSSH daemon running all the time and can serve as a nice sshd backup (if someone kills it by mistake for example), it's however slower.

Running SSH as a daemon

The other method of running OpenSSH is as a standalone daemon. This is the preferred and faster mode. On RedHat like distributions you can do:

service sshd (start|stop|restart|reload|condrestart|status)

in order to start, stop, restart (after configuration changes), reload (like the previous, but without stopping and starting, just reloading the configuration), restart if already running and get the daemon's running status - running, stopped, dead / who killed my daemon! :-) /. To make the daemon run on restart you can use 'chkconfig sshd on' and the opposite - 'chkconfig sshd on'. You can also tune the runlevels on which you want the SSH daemon running, but that's out of this article's scope, consult your manual if you run non-RedHat distribution.

Configuring

The next step is to configure your SSH server. OpenSSH is secure by default, but there's always room for enhancements, so let's get our a bit hands dirty.

All configuration files are located in /etc/ssh (if you installed from source, did you... why?). There are 9 files most likely: moduli - don't touch this one unless you know what you are doing (I don't), ssh_config - general client configuration, sshd_config - the server's configuration and ssh_host_dsa_key, ssh_host_dsa_key, ssh_host_dsa_key.pub, ssh_host_key, ssh_host_key.pub, ssh_host_rsa_key, ssh_host_rsa_key.pub - all public and private keys - these get generated at the first daemon start and I don't look at them as regular configuration files exactly. If you change your server's name/IP address you need to remove these and let sshd regenerate them ('service sshd start' does this, for me at least). If you regenerate these files in most setups you'll have to also touch your ~/.ssh/known_hosts file to make your SSH client forget the old host keys. I'll stop on sshd_config and then ssh_config, let's see what's in there!

sshd_config

As I said, this is the server's configuration file - the one you should first focus your attention on. Before you change anything I suggest you install the ssh-backup xinetd service and start playing with it's configuration file (or use it as a backup, better not). This way you don't risk killing your SSH server with a bad configuration (and having to run to the other end of the state if the server is there). If you follow my advice, please do, you will make all changes first to sshd_backup_config, test them by

ssh your_username@your.remoteserver.test.com -p 2022

(make sure your server allows you to connect to that port - firewall, tcpwrappers, SELinux) and copy them to sshd_config after everything is nice and dandy. The server by default will ask you for your password and let you in (which we are going to change in a little while, stay tuned). Because the backup sshd (it's being run from xinetd) will read it's configuration at start time (the time you connect) you just edit the configuration file (sshd_backup_config, remember?), save and connect to test - restart nothing.

For the rest of the time I'll suggest you open 3 consoles (3 tabs on 'konsole'(KDE) or 'gnome-terminal'(GNOME), start three 'xterm's or login on three text consoles - you choose).

The first one we'll use to watch the info/error messages from your server - login there and 'tail -f /var/log/messages):

[mr700@laptop:~]$ ssh mr700@remote
mr700@remote's password: <type your password here, nothing is echoed back>
Last login: Tue Sep 11 21:58:32 2007 from laptop
# if you use sudo:
[mr700@remote:~]$ sudo tail -f /var/log/messages /var/log/secure
# if you don't use it or know what am I talking about ;-)
[mr700@remote:~]$ su -
Password: <type your root password for the remote machine, nothing echoed>
[mr700@remote:~]# tail -f /var/log/messages /var/log/secure
# both ways you'll see something like:
==> /var/log/messages <==
Sep 11 21:59:30 remote last message repeated 3 times
...

==> /var/log/secure <==
Sep 11 23:19:26 mr700 last message repeated 7 times
...

Leave this console alone, we will look for errors in the configuration, denies and logins later on here.

On the second console we'll edit 'sshd_backup_config' and copy changes back to sshd_config when we are happy with the result. NB: Whatever port or ListenAddress address you specify in the backup configuration does not matter - these are xined's job. Leave your favorite text editor (mcedit, vi, joe or whatever you like) with this file open.

The third console will be for tests - just type

ssh your_username@your.remoteserver.test.com -p 2022

and don't hit enter just yet. We'll use it in a second.

ssh_config (~/.ssh/config)

In /etc/ssh/ssh_config the things you can configure can be configured per user in his ~/.ssh/config file. The global one can however save some time if you have lot's of users and machines from which they (you) use SSH.

Playing with the fire

So, let's start with something easy - turn off reverse DNS lookups for your daemon. Older OpenSSH servers had no such option, but we are running a recent one, right? I'll use this basic example to show you how to check for errors in your configuration and correct them safely. Scroll down in your 2-nd console and find the line that looks like '#UseDNS yes' and type 'UseDNS nope' (yes, type exactly that - nope) after it (or at the end of the file if there's no such one). The '#' character is a comment in OpenSSH's configuration files, so comment out the old line if needed. Let's try this (console 3) and see the result on console 1 (the log): On console 3:

[mr700@laptop:~]$ ssh mr700@remote -p 2022
ssh_exchange_identification: Connection closed by remote host
[mr700@laptop:~]$ _

Oopss... what happened? Take a quick look at console 1:

Sep 11 23:43:38 remote xinetd[855]: START: ssh-backup pid=13428 from=10.0.0.1
Sep 11 23:43:38 remote sshd[428]: fatal: /etc/ssh/sshd_backup_config line 113: Bad yes/no argument: nope
Sep 11 23:43:38 remote xinetd[855]: EXIT: ssh-backup status=255 pid=428 duration=0(sec)

A-ha! As you have guessed - nope is not a bad argument to for UseDNS. Now change it to 'no', save the file and try again. After logging in logout (press Ctrl-D) and you'll see something like

==> /var/log/messages <==
Sep 11 23:52:07 remote xinetd[855]: START: ssh-backup pid=434 from=laptop

==> /var/log/secure <==
Sep 11 23:52:11 remote sshd[434]: Accepted password for mr700 from laptop port 57680 ssh2
Sep 11 23:52:11 remote sshd[434]: pam_unix(sshd:session): session opened for user mr700 by (uid=0)

==> /var/log/messages <==
Sep 11 23:52:13 remote xinetd[855]: EXIT: ssh-backup status=0 pid=434 duration=6(sec)

==> /var/log/secure <==
Sep 11 23:52:13 remote sshd[434]: pam_unix(sshd:session): session closed for user mr700

in your logs. Nice and dandy, just like I promised - copy this back to your ssh_config and check your main SSH daemon is happy about it (it will) restarting it. Even if it fails your backup variant is working - you can clean the mess. Always keep one of them working to be able to rescue the other. It's always a good idea to check if your services complain about something. From now on, I'll leave this check and all diagnostics to you (I'll have to show you the same text all the time).

The advantage of using no reverse DNS lookups is faster connection time (the daemon will not wait for DNS queries) and you'll always know the IP address, not some hostname that may or may not resolve back to the same IP address. This can be a problem if you use dynamic DNS and your DHCP clients update the DNS records, but I for one always use 1:1 mac-ip mapping in my DHCP server and you can always check your logs to see who used that IP at that time.

Last thing I want to explain before we start seriously. When I say change option X to Y that means locate X in the configuration file and comment it out (if it's there), just change it or add it if it's not there. It's also a good idea to keep the old working configuration somewhere while changing it.

Securing the OpenSSH daemon

Let's see what can we do to increase our SSH daemon's security. Don't forget that security is not a product, service, configuration, restrictions - it's an ongoing process all the time and it includes user education. If, for example, your users write down their passwords/passphrases on small stickers on their monitors/keyboards or share them you've lost this battle. Never thought that's possible? Go look in your office... look careful! Shocked?

Remember that if your security measures take most of your users' time and energy you've lost again. Well, if they can't even remember one password... then you can safely report them for replacement. :-]

Always explain to your users why these measures are needed - they may not understand every word you say, but the can get the right impression from your tone. Practice opera singing 1 hour every day for to make the tone trick to work. :-D

And last, but not least, keep in mind that this process starts from you, yes, I mean you! You also have to change your passwords/passphrases often enough (not every day, every say 3-12 months is a good period).

Disabling passwordless logins

The first and most obvious security hole you can open is to allow anyone login without password. Even if his/her username looks like password, it gets logged all over your server logs, it ca be guessed from his/her personal page in apache and so on, and so on... But you already know all this, don't you? So, let's just do it. Check if you have 'PermitEmptyPasswords yes' in your configuration, the default is no. Just to make sure add it if it's not there and make it 'PermitEmptyPasswords no'. Now you will sleep better...

Disabling root login

Some may say that this measure is a bit draconian, but it can increase security (everyone is trying root login assuming it's there). If you don't want or just can't completely disable remote root logins, then you can at least deny remote root logins with passwords. This is done with

PermitRootLogin yes
PermitRootLogin without-password
PermitRootLogin forced-commands-only
PermitRootLogin no

Don't put them all - select one (I know, that :-) you know). The first one allows root login unconditionally (PermitEmptyPasswords can still deny your access), the next one disables password authentication for root logins, the third one allows only forced (only specified) commands and the last one denies all root logins. Don't forget to test and verify your changes.

Using SSH version 2 protocol only

Restricting your ssh to SSH version 2 protocol only is a must. It does not causes any troubles these days, almost everybody supports it. SSH version 1 is not as secure and you risk man-in-the-middle attacks. The setting is simple - 'Version 2', that's all.

Restricting ssh access to users/groups

Your system can have multiple users, but maybe not all of them must be allowed remote logins. This step will also protect you from configuration mistakes like setting password for your apache user. You can allow only specific user(s) to login or restrict them by group. Let's say you want only users from the 'wheel' group to be able to login in this system. Make sure your user is member of the group on the remote system

[mr700@remote:~]$ id
uid=700(mr700) gid=700(mr700) groups=10(wheel),20(games),100(users),499(fuse),700(mr700), context=user_u:system_r:unconfined_t

my user will do in this case. Next thing is to change/add 'AllowGroups wheel'. You can also use 'DenyGroups', 'AllowUsers' and 'DenyUsers' and give any number of users/groups separated by spaces. "The allow/deny directives are processed in the following order: DenyUsers, AllowUsers, DenyGroups, and finally AllowGroups" (quote from the manual).

Using IP address restrictions

Restricting access to your SSH server is very good idea (if possible, if you travel every day and login from different countries it's not an option or you have to do some VPN tricks). It's also usability vs security here, but at least for some servers it's applicable. There are three different ways to accomplish this:

You can configure your SSH server to bind to internal IP addresses only. Add 'ListenAddress' directives for every interface (it's IP address) you want your server to bind to. 'AddressFamily' can be used to restrict it to IPv4/IPv6 only communication too. Don't forget to test and keep in mind your backup SSH daemon ignores these.

The next method is universal - use your firewall - iptables. Writing firewalls (especially good ones) is an art, so I'll just leave this to you and will just mention you can combine it with the next method :-)

If your SSH daemon is compiled with tcpwrappers support (most likely) you can use your /etc/hosts.allow and /etc/hosts.deny files to restrict access to it by IP. It's good to combine iptables with tcpwrappers, just don't forget what you did or you can have some sleepless nights one day...

DenyHosts

There are a number of external tools than can change your firewall/tcpwrappers/configurations in order to block most offenders (or script kiddies for that matter). This program looks for failed ssh login attempts and after configurable number of attempts blocks access to a given IP address for also configurable amount of time.

Another feature is that it can synchronize it's database with a central server, allowing you benefit from others' databases and uploat your findings there. This feature can probably be a DOS vector, but I have a backup VPN that is explicitly allowed and haven't dug too deep here.

The last thing I like about this little program is that it uses tcpwrappers and does not mess with my firewall. That's the place I explicitly allow my VPN IP addresses also (tcpwrappers). I also use non-standard port number, which reduces it's hits to just a few per month on my systems.

Using public keys and ssh-agent

OpenSSH offers variety of authentication methods, but the general rule is the less you allow, the more secure your server is. If a vulnerability is discovered in say the password authentication code and your server does not allow it (besides from the fact it will be scary thing to read in the news), you will be on the safe side. On the other side, the most secure server is the one locked in a big safe, so it's security vs usability here. To show you how this is done, let's limit our server to public keys only. For this to work, I'll have to first introduce you to some new heroes here:

ssh-keygen

In order to use ssh key we need to have one, right? The key is generated with ssh-keygen, here's an example session showing how this is done:

[mr700@laptop:~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/mr700/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/mr700/.ssh/id_rsa.
Your public key has been saved in /home/mr700/.ssh/id_rsa.pub.
The key fingerprint is:
a1:23:a4:64:86:3a:bd:40:38:2a:e7:f2:cc:fe:bc:cc mr700@laptop

Remember that id_rsa is your private key - no one should ever touch/see this file except you. id_rsa.pub is your public key - you can even post this one on your website. If someone adds the later to hist ~/.ssh/authorized_keys2 file you will be able to login there. The main idea however, is not to lure someone :-) in doing this, but to put it on all remote machines you are going to login. Check the permissions of your ~/.ssh directory and private key(s) /you can have more than one key/. These should look like this:

[mr700@laptop:~]$ ls -ld .ssh .ssh/id_rsa*
drwx------ 4 mr700 mr700 4096 11 Sep 23,51 /home/mr700/.ssh
-rw------- 1 mr700 mr700 1743 22 Aug  4,26 /home/mr700/.ssh/id_rsa
-rw-r--r-- 1 mr700 mr700  405 22 Aug  4,26 /home/mr700/.ssh/id_rsa.pub

If the directory ~/.ssh or the private key ~/.ssh/id_rsa have less restrictive permissions ssh will refuse to use them.

You can leave your passphrase empty and it will work just fine, but please don't do so. If someone steals your key and there's no passphrase protecting it... you are lost, you have been warned.

Now, let's copy your new key to the remote machine and try it:

[mr700@laptop:~]$ scp .ssh/id_rsa.pub remote:
mr700@remote's password:
id_rsa.pub                    100%  405     0.4KB/s   00:00
[mr700@mr700:~]$ ssh remote
mr700@remote's password:
Last login: Tue Sep 11 23:52:11 2007 from laptop
[mr700@remote:~]$ mkdir .ssh
[mr700@remote:~]$ chmod 700 .ssh
[mr700@remote:~]$ chmod 600 id_rsa.pub
[mr700@remote:~]$ ls -dl .ssh id_rsa.pub
-rw------- 1 mr700 mr700  405 12 Sep  6,11 id_rsa.pub
drwx------ 4 mr700 mr700 4096 12 Sep  6,10 .ssh
[pts/11 mr700@mr700:/home/mr700]$ mv id_rsa.pub .ssh/authorized_keys2

Logout (Ctrl-D or just logout) and try logging in again with your key:

[mr700@laptop:~]$ ssh remote -x -v
OpenSSH_4.5p1, OpenSSL 0.9.8b 04 May 2006
...
debug1: Connection established.
debug1: identity file /home/mr700/.ssh/id_rsa type 1
...
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/mr700/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: PEM_read_PrivateKey failed
debug1: read PEM private key done: type 
Enter passphrase for key '/home/mr700/.ssh/id_rsa':
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
...
Last login: Wed Sep 12 06:20:37 2007 from laptop
[mr700@remote:~]$ _

I have removed some of the debug info (being shown because of the -v switch) leaving only the relevant one (it's ~ 40 lines). You can see that the remote server says I can use publickey key password authentication, my client says it can supply /home/mr700/.ssh/id_rsa, the server says it would accept it, my client tries to read the private key and finds out a passphrase is neede. After I typed my passphrase my client can 'send' (it's not sending the exactly the key) it and the connection is established. This for now is more secure, but is the same amount of typing for you, so let's see what else does OpenSSH have in store for us.

ssh-agent and ssh-add

You can run ssh-agent as a daemon (well, personal daemon) and add keys to it with ssh-add. This way you will type your passphrase once and connect to any host any number of times without retyping it again. If you are running X11 (you aren't reading this with lynx, are you?) most likely someone did the first part for you (running the agent and exporting all needed data, if not - check the manual). You can verify this with 'echo $SSH_AGENT_PID' and 'echo $SSH_AUTH_SOCK' - if these are not empty set you have ssh-agent running. Fire ssh-add:

[mr700@laptop:~]$ ssh-add
Enter passphrase for /home/mr700/.ssh/id_rsa:
Identity added: /home/mr700/.ssh/id_rsa (/home/mr700/.ssh/id_rsa)
[mr700@laptop:~]$ _

This approach, besides being much more nice to work with, is even more secure - the less you type your passphrase, the harder eavesdropping it is for everyone is. When you are done ssh-ing here and there either do 'ssh-add -D' to make ssh-agent forget your keys or make sure no one touches your consoles. Your passphrase, just like your password, must be changed after some time. Apply common sense here :-)

Testing it

That's the easy part, go to your 3-rd console and try logging in again - should just work(tm).

Another thing you can do with keys is, tell ssh-agent to forward your authentication agent data to the remote host. This means, that if you can login to remote1 with your keys you can go on from there and login to another machine (remote2) with the same keys, without having to copy them on remote1. The switch doing this is '-A' (ssh -A you@remote1). This can be done in ~/.ssh/config or /etc/ssh/ssh_config with 'ForwardAgent yes'.

Disabling all other authentication methods

The last step here is to disable all/most other authentication methods - the less, the better. As you have seen, using '-x' in the debug lines there was a line stating all allowed authentication methods that you can continue connecting with. Make sure 'RhostsRSAAuthentication no', 'HostbasedAuthentication no', 'IgnoreRhosts yes' and 'GSSAPIAuthentication no' are not set or set to these values (they are the default setting). Set 'PasswordAuthentication' to 'no' and you should be ready (man is your friend if not).

NB: Check carefully you still can connect with the methods left before applying the change to the main daemon!

Using non-standard high port number

Well, as lame as it is (changing the port number I mean), this old dirty trick will reduce the size of your logs considerably - most Internet worms only try port 22. Take good care not to ask for a port in your dynamic port range or you may one day (after restart) end up with no ssh server. To check your local dynamic port range type the following:

[mr700@remote ~]$ cat /proc/sys/net/ipv4/ip_local_port_range
32768   61000

This means I should not select ports in the range [32768..61000], both these numbers included. You can't test this setting with your backup ssh server, so don't touch the backup, just use the daemon for the test and keep the backup running in case of disaster.

Adjusting clients' configuration

If you use non-standard port numbers that quickly becomes annoying (you have to type the port every time). This can be easily avoided. Remember ssh_config? It can be used to specify this ports per host basis for all users like this:

Host remote
        Protocol 2
        Port 2022
        User mr700
        HostName 10.0.0.2
        ForwardX11 no
        # ForwardX11Trusted yes
        ForwardAgent yes
        Compression no
        # CompressionLevel 0
        GSSAPIAuthentication no

in users' ~/.ssh/config or system-wide in /etc/ssh/ssh_config. Please don't use my port numbers, select your very own for better results (and these are not the ones I use also).

Using port-knocking techniques

This is relatively new technique and it's more secure than the non-standard port solution, but a bit harder to implement in some situations. See http://en.wikipedia.org/wiki/Port_knocking for general definition. Port-knocking can even be implemented right into your iptables firewall without any extra tools. Check http://www.shorewall.net/PortKnocking.html to see how this can be done with shorewall too (shorewall generates iptables rules after all). There's also information how to limit the allowed connection rate. DenyHosts can take some time to kick in, iptables do their job instantly. Port-knocking is not SSH specific, you can use it for virtually any service, just make sure it's not your only defense for it (SSL).

known_hosts, ssh_known_hosts

Maybe I should have started wit this one. Even most administrators, me included in the general case - shame on me, don't check their SSH servers' public key or at least it's fingerprint when connecting for the first time. OpenSSH does a nice job here remembering this information (it asks you when you first connect to a new host) and warning you about man-in-the-middle attacks when these keys don't match later, but that works assuming you was not hacked the first time you connected to the server.

You can/should copy your servers' public keys physically, by hand, using USB Pen Drive for example, to your machine. This will ensure no one played main-in-the-middle on you. Why do you think aliens are little green man? Because they all get green while copying their ssh host keys to all planets all over the universe... :-) The known_hosts' file format is simple - hostname,hostname2,hostip1,hostip2, space, the public key... well, sort of - check 'man 8 sshd' for details. The public key file is obtained from /etc/ssh/ssh_host_rsa_key.pub, the first part is up to you to compose. The system-wide file is '/etc/ssh/ssh_known_hosts', users can manage (ssh does this by itself unless instructed otherwise) in '~/.ssh/known_hosts'.

You might find the following link of interest on this subject - http://itso.iu.edu/Hashing_the_OpenSSH_known__hosts_File.

Securing the client

While securing your server, why not secure your clients too? Most obvious thing you can do is to disable X11 forwarding by default if you don't need it. This can be done per host or globally in '/etc/ssh/ssh_config' or your own - '~/.ssh/config'. Because it's not as dangerous to do (it's on the local machine) and you have seen an example how to use these while we were discussing changing the standard port number, I'm leaving the rest to you - 'man 5 ssh_config'. Nice playing.

Other uses of OpenSSH

As I mentioned earlier, SSH can do X11 forwarding (DXPC can help a lot here I'm told - see http://www.vigor.nu/dxpc/), secure rdesktop/VNC and support NX connections, create SOCKS4/5 proxy, tunneling... but my limit of characters per article is almost reached (and maybe your patience) , so I'll leave these for the next one. Here I just want to mention that you can disable most/all of them, but this will not make your server more secure. To make this restrictions effective, you'll have to implement draconian security on the host (remember the usability vs security part?) or your users will be able to do the same things with more effort...

What else would you like to read there, I'd like to hear from you, mostly positive comments of course. :-]