I'll be the first to admit my security has room for improvement. Until last year, I was reusing passwords intermixed with a terribly simple mnemonic. Until a few months ago, my phone and computer were totally unencrypted. I've been fighting the change because it's scary. I'm also very lazy and have been dreading the extra work involved with good security. I've put off updating SSH credentials for about two years now for that exact reason.

I decided to at least pretend like I was doing something about that this morning. It's a gargantuan task: managing many keys with many passphrases on many machines with so much typing involved. Rather than actually sitting down to update things, I poked around man pages and Stack Overflow for a bit, avoiding more than anything else. Somehow I managed to put together a halfway decent solution that, so far, seems to remove almost all the heavy lifting.

Requirements

If you've never used KeePass or KeeAgent, start here. I've written about the two together before.

On the other hand, it should be possible to set up most of this with the vanilla ssh-agent or something more opinionated like gnome-keyring-daemon or kdewallet5. Taking KeePass and KeeAgent out of the equation, in my opinion, generates way more work, but YMMV.

Note

KeePass hasn't been thrilled with i3 and my theme choices. I apologize for the strange layouts. Unless you decide to go and change everything it doesn't normally look this bad. I was too excited about the solution to go and undo all my tweaks.

Problem

Simply put, managing a ton of keys is a serious pain. If, like me, you've never really delved into the myriad ways to beef up your settings, you'll hit a few snags:

  • You have to manually add a ton of keys to ssh-agent, which means typing passphrases for days when your power goes out, killing the longest continuous uptime you've had in months
  • You have to juggle keys in the agent, as most servers have a fairly low retry count (default is, I believe, six)
  • You have to manually enter the passphrases for anything specified in your config that's not active in your agent, which seems like a waste of time if you're just going to have to juggle anyway

Solution

Those statements, built on experience and prevailing wisdom, touch on some pretty major annoyances. However, they all suffer from a perspective problem. They begin with the assumption that a user has many keys and connects to many machines. That's very true. But it ignores a very important detail: while there are many possible permutations of key and machine, the number of successful permutations is much smaller and (in theory) proper matchings are already known.

ssh_config defines a vehicle to specify proper matchings. Providing an IdentityFile (or several) per host establishes a match. IdentitiesOnly prevents ssh from trying anything not specified in the config or CLI, effectively sidestepping the retry issue. IdentityAgent provides an agent failsafe (or multiple agents in tandem). KeePass and KeeAgent glue everything together with remembered credentials, making the config much more powerful.

Assuming ~/.ssh/some_key is in KeePass and available to KeeAgent, this simple config is enough to get you started:

~/.ssh/config
1
2
3
4
5
6
Host simple_name
HostName fqdn
User remote_user
IdentitiesOnly yes
IdentityFile ~/.ssh/some_key
IdentityAgent SSH_AUTH_SOCK

Example

All of this will make much more sense after a good example. I'm going to build everything from scratch and try to illustrate the pain points I believe I've mitigated.

Generate Keys

To demonstrate something closer to an ideal environment, I've generated ten keys. They're all ridiculously weak and shouldn't be used in production (passphrase was passed via the CLI, which should be enough warning).

$ ssh-keygen \
-t rsa \
-P password03 \
-C user03@host \
-f dummy_key_03
Generating public/private rsa key pair.
Your identification has been saved in dummy_key_03.
Your public key has been saved in dummy_key_03.pub.
The key fingerprint is:
SHA256:SxGPZn/19LYMIEA05z+4sezZ9ruBG1cQdaaBf0sr5Tc user03@host
The key's randomart image is:
+---[RSA 2048]----+
| o=.. oo.o|
| =+ . .+.|
| =o.. oo .|
| o o+ ..o=o|
| So.o..+o*|
| ...+.o.+Eo|
| .+ o o.oo|
| . o.+ . |
| o.o.+o |
+----[SHA256]-----+

Populate KeePass

With the keys in hand, I populated the KeePass database next. Each entry's password is the passphrase to the key, and the keys themselves are attached to the entries.

loaded-db

The easiest way to make things work is to allow KeeAgent to use each entry. If you don't, you'll have to manually add the key later (which has its uses).

keeagent-enabled

All told, this is about seven more active keys than I'm used to having around. It's a strange feeling, to say the least.

all-in-keeagent

Create Environment

To make things as simple as possible, I created a vagrant box that will act as the remote. I did a few things:

  • Created dummy_user
  • Assigned dummy_key_10 (the last in the list) as dummy_user's primary key (also authorized_key)
  • Removed password access (forcing a key exchange)
  • Lowered the number of attempts to three

vagrant exposes its own ssh_config via ssh-config, which we'll use to to access the box and later as a template for our own.

$ vagrant ssh-config
Host default
HostName 192.168.121.150
User vagrant
Port 22
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile ...
IdentitiesOnly yes
LogLevel FATAL

Easy Failure

We're going to try to connect with all the keys active.

$ ssh-add -l
2048 SHA256:7Fd0YO1OENizISLV+rzBHc+KHpsDKnfkZoPEOKNoGt4 user01@host (RSA)
2048 SHA256:/alv7iswhB0YMgYtUYbVu4UTHCLpqa7M/o7PXBf5NJk user02@host (RSA)
2048 SHA256:7FBCSXS7hWUWV7PxRvw65PytkoF65gh/b4vJU49jBNU user03@host (RSA)
2048 SHA256:CL4UsKgCscVv3WyFY0RgiZfRrfOPI+etatN779C7fX0 user04@host (RSA)
2048 SHA256:IWSIHh8A3ApK0s0SaxIxeyRpSTmoMjMFW827iPvc3OE user05@host (RSA)
2048 SHA256:E3ovjpFD524ZoYmzUO/ucBDAfGAyeP7jzwlaLqi0Mi0 user06@host (RSA)
2048 SHA256:Qh/Nss0znxMwpTGlOqpxacsrj8in3xYv4A8bcRhH/Ik user07@host (RSA)
2048 SHA256:md1KsOzJ/BsYtZBgFA5PxhR4eyEfUwFPdoTurhHDEao user08@host (RSA)
2048 SHA256:SEu3eaPbHPRgyp/7lpkbT1Wpcv2oZ45vOWshTgXiZp0 user09@host (RSA)
2048 SHA256:3nzgRuofhgfPHgfQQERk0b1TmRLufV0ppWwixojFjlk user10@host (RSA)
$ ssh dummy_user@192.168.121.150
Received disconnect from 192.168.121.150 port 22:2: Too many authentication failures
Disconnected from 192.168.121.150 port 22

Quick and unsurprising.

Easier Success

We can just as quickly get in with a simple config file.

~/.ssh/config
1
2
3
4
5
6
Host vagrant-box
HostName 192.168.121.150
User dummy_user
IdentitiesOnly yes
IdentityFile /path/to/dummy_key_10
IdentityAgent SSH_AUTH_SOCK

The key path is fairly important. ssh attempts to load the key, sees KeeAgent already has it available, and moves on. Without a local copy of the key, there's nothing for ssh to go off of. (I think; this is an educated guess using the debug logs. Might not be 100% accurate.)

$ ssh vagrant-box
Last login: Sun Feb 25 17:09:45 2018
[dummy_user@localhost ~]$ exit
logout
Connection to 192.168.121.150 closed.

Recap

By themselves, ssh_config and KeePass/KeeAgent are very powerful tools. Together they mitigate the need to juggle keys and constantly enter passphrases. IdentityFile, IdentitiesOnly, and a little bit of setup will make using more than one key a painless endeavor. Until you have to update them all...

Full Scripts

These are in the repo but I also wanted to lay out everything here.

keygen

keygen
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
#!/usr/bin/env python
# coding: utf8

"""This file provides a script to generate dummy ssh keys"""

# pylint: disable=misplaced-comparison-constant

from os import makedirs
from os.path import abspath, dirname, join
from shutil import rmtree
from subprocess import check_call

KEY_COUNT = 10
SCRIPTS_DIR = abspath(dirname(__file__))
KEY_DIR = abspath(join(SCRIPTS_DIR, '..', 'keys'))
KEEPASS_DIR = abspath(join(SCRIPTS_DIR, '..', 'keepass'))
DATABASE_PATH = join(KEEPASS_DIR, 'NewDatabase.kdbx')


def rebuild_key_dir():
"""Erase and remake key dir"""
rmtree(KEY_DIR, True)
makedirs(KEY_DIR)


def wipe_database():
"""Wipes all KeePass entries"""
check_call(
[
'kpscript',
'-c:DeleteAllEntries',
DATABASE_PATH,
'-pw:',
]
)


def create_single_key(index):
"""Creates a key for a given index"""
check_call(
[
'ssh-keygen',
'-t', 'rsa',
'-P', "password%02d" % index,
'-C', "user%02d@host" % index,
'-f', join(KEY_DIR, "dummy_key_%02d" % index)
]
)


def update_database(index):
"""Adds the entry to the database"""
check_call(
[
'kpscript',
'-c:AddEntry',
DATABASE_PATH,
'-pw:',
"-UserName:dummy_key_%02d" % index,
"-Password:password%02d" % index
]
)


def create_keys():
"""Creates all the necessary keys"""
for index in range(KEY_COUNT):
create_single_key(index + 1)
update_database(index + 1)


def cli():
"""Runs everything"""
rebuild_key_dir()
wipe_database()
create_keys()

if '__main__' == __name__:
cli()

Vagrantfile

Vagrantfile
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
config.vm.box = "fedora/27-cloud-base"

config.vm.provision "shell" do |shell|
shell.privileged = true
shell.inline = <<-SHELL
useradd dummy_user
echo dummy_password | passwd dummy_user --stdin
mkdir -p /home/dummy_user/.ssh
chmod 'u=rwx,go=' /home/dummy_user/.ssh
cp /vagrant/keys/dummy_key_10* /home/dummy_user/.ssh/
chown -R dummy_user:dummy_user /home/dummy_user
cp /vagrant/keys/dummy_key_10.pub /home/dummy_user/.ssh/authorized_keys
chmod 'u=rw,go=' /home/dummy_user/.ssh/authorized_keys
sed -i \
-e 's/PermitRootLogin/#PermitRootLogin/g' \
-e 's/#MaxAuthTries 6/MaxAuthTries 3/g' \
-e 's/PasswordAuthentication yes/PasswordAuthentication no/g' \
/etc/ssh/sshd_config
systemctl restart sshd
SHELL
end
end