/ Let's Encrypt from Start to Finish

Let's Encrypt from Start to Finish: Tuning with OpenSSL

This is the third in a series of several posts on how to do way more than you really need to with Let's Encrypt, certbot, and a good server. I use all of these things regularly but I've never taken the time to take them apart, look at how they work, and spend hours in Google trying in vain to figure out how to put them back together. It was inspired by a disturbing trend of ISP privacy violations and the shocking regulatory capture of the US Federal Communications Commission.

This post sets up all the backend security logic (minus headers) to harden Nginx or Apache. I've tried to provide an explanation of each component and good values to use (or the means to create your own). If you don't have OpenSSL, most of this is meaningless.

The Series so Far

  1. Overview
  2. First Steps
  3. Tuning with OpenSSL
  4. Useful Headers
  5. Generating and Testing a Cert
  6. Automating Renewals

Things that are still planned but probably not soon:

  • Updating OpenSSL
  • CSP Playground
  • Vagrant Examples (don't hold your breath)

Code

You can view the code related to this post under the post-03-openssl-tuning tag. If you're curious, you can also check out my first draft.

Note

I'm testing out some new tooling. This will be wotw-highlighter's shakedown run. Let me know what you think about the syntax highlighting! I'm pretty excited because (no whammies) it should work well in AMP and normally.

I wrote the majority of the Apache examples with httpd in mind, i.e. from a RHEL perspective. If you instead use apache2, most of the stuff should still work, albeit maybe just a little bit different.

Primary Security Reference

I'll be using the Qualys suggested configuration to set this up. Most of this stuff is explained elsewhere on the internet. I wanted to grok the whole process, so I wrote it up.

If you're reading this more than, say, a month or two from its publication date, I'd strongly urge you to follow the documentation links to find the most current algorithms, best practices, and so on. Even if my minutae is current, you should always check sources when security is involved.

Primary Config File

This creates a single file to hold the common config.

As I said before, I like /etc/<server>/common/, YMMV.

Nginx

$ sudo touch /etc/nginx/common/ssl.conf

Apache

So this probably won't work without some TLC. Apache differentiates global vs scoped config, and some of the things I mention only work in one or the other. The official docs state scope per directive and I've tried to match that. However, I'm going to pretend like it will work without issue and hope no one says anything.

$ sudo touch /etc/httpd/common/ssl.conf

You'll also need to ensure you've got the right modules installed and running. Depending on your server's distro and the version of Apache you're running, installing and enabling modules is done differently.

$ which a2enmod && echo "apache2" || echo "httpd"
RHEL is usually httpd
Debian is usually apache2
  • If you're running httpd, enable them by editing /etc/httpd/conf.modules.d/00-base.conf (or another file there; you might have to grep them out).
  • If you're running apache2, enable them via a2enmod.

You'll need these modules:

  • mod_rewrite
  • mod_ssl
  • mod_socache_shmcb for any caching (sessions, stapling)
$ eval "$(which apachectl && echo apachectl || echo httpd) -M" | grep -E "rewrite|shmcb|ssl"
rewrite_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)

You might actually have to install additional external packages depending on how you get Apache, e.g. mod_ssl on RHEL systems.

Specify Allowed TLS Versions

Qualys says v1.2 is the only secure version. v1.3 is only a draft, so including it might be odd. If you're truly desperate, v1.1 isn't too bad. Don't forget that Qualys writes the benchmark, so if you ignore that advice, your rating will take a hit.

Nginx

1
ssl_protocols TLSv1.2;

Apache

1
SSLProtocol -all +TLSv1.2

Generate a List of Good Ciphers

You might check the current list to make sure this is up-to-date. You can also shorten this list; I was curious how it was built.

/save/the/qualys/list/somewhere
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES128-SHA
ECDHE-ECDSA-AES256-SHA
ECDHE-ECDSA-AES128-SHA256
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-SHA
ECDHE-RSA-AES256-SHA
ECDHE-RSA-AES128-SHA256
ECDHE-RSA-AES256-SHA384
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES128-SHA
DHE-RSA-AES256-SHA
DHE-RSA-AES128-SHA256
DHE-RSA-AES256-SHA256

We can use grep to search with a pattern from a -file, composed of newline-separated -Fixed strings, where each pattern matches the entire line (-x). All we need is the available ciphers. openssl ciphers returns a colon-separated list, so we can pass it through translate before searching it.

$ openssl ciphers
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:DH-DSS-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DH-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DH-RSA-AES256-SHA256:DH-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DH-RSA-AES256-SHA:DH-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:DH-RSA-CAMELLIA256-SHA:DH-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:DH-DSS-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:DH-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DH-RSA-AES128-SHA256:DH-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DH-RSA-AES128-SHA:DH-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DH-RSA-SEED-SHA:DH-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:DH-RSA-CAMELLIA128-SHA:DH-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:PSK-AES128-CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SRP-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:SRP-3DES-EDE-CBC-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DH-RSA-DES-CBC3-SHA:DH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA
$ grep -Fx -f /save/the/qualys/list/somewhere <( openssl ciphers | tr ':' '\n' )
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA
ECDHE-ECDSA-AES256-SHA
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-SHA256
DHE-RSA-AES256-SHA
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA256
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA
ECDHE-ECDSA-AES128-SHA
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-SHA256
DHE-RSA-AES128-SHA

If you didn't get any results, you should probably spend the few days necessary to move everything to a platform at least from the last decade. Or maybe openssl didn't get set up correctly.

Unfortunately, this doesn't retain the order Qualys uses, and I've yet to figure out a good way to maintain the original order. A standard bash approach would be to commpare the two lists, but comm expects the lists to be sorted.

$ comm -12 /save/the/qualys/list/somewhere <( openssl ciphers | tr ':' '\n' )
comm: file 1 is not in sorted order
ECDHE-RSA-AES256-GCM-SHA384
comm: file 2 is not in sorted order
ECDHE-RSA-AES256-SHA384
$ comm --nocheck-order -12 /save/the/qualys/list/somewhere <( openssl ciphers | tr ':' '\n' )
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384

Brace expansion doesn't reduce the size by much, so I gave up after a few hours on that tangent. However, the important thing is that you now know what ciphers are available for you to use that also have the Qualys stamp of approval.

Nginx

I don't think specifying all these is a great idea, but I actually have no idea.

1
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256;

Apache

I don't think specifying all these is a great idea, but I actually have no idea.

1
SSLCipherSuite "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256"

Specify ECDHE Curve

Qualys suggests using a specific elliptic curve for ECDHE. Elliptic Curve Cryptography, like other algorithms, has both secure and insecure methods. Also like other algorithms, there are a few common, widely used components. Qualys recommends secp256r1 or P-256. Coincidentally, that curve has most likely been backdoored by government agencies for years and has even been put down by government agencies more recently.

I mention that because there's basically no way to protect yourself from actors with superior tech. No matter what you do, there's always someone with more money and decades of classified algebra and combinatorics to keep you grounded. However, unless some of that knowledge has been put to code and leaked recently, most of the cryptography mentioned here will probably prevent wardriving script kiddies from messing with you or your users for a few years.

If you'd like to use something a bit more secure, check out SafeCurves. Like before, we'll need to check what's available to us with the installed version of OpenSSL.

$ openssl ecparam -list_curves
secp112r1 : SECG/WTLS curve over a 112 bit prime field
...
secp521r1 : NIST/SECG curve over a 521 bit prime field
prime192v1: NIST/X9.62/SECG curve over a 192 bit prime field
...
prime256v1: X9.62/SECG curve over a 256 bit prime field
sect113r1 : SECG curve over a 113 bit binary field
...
sect571r1 : NIST/SECG curve over a 571 bit binary field
c2pnb163v1: X9.62 curve over a 163 bit binary field
...
c2tnb431r1: X9.62 curve over a 431 bit binary field
wap-wsg-idm-ecid-wtls1: WTLS curve over a 113 bit binary field
...
wap-wsg-idm-ecid-wtls12: WTLS curvs over a 224 bit prime field
brainpoolP160r1: RFC 5639 curve over a 160 bit prime field
...
brainpoolP512t1: RFC 5639 curve over a 512 bit prime field

Comparing my list to SafeCurves yields zero safe curves. That's because OpenSSL has rolled most of those into v1.1, which isn't in stable channels yet (and probably won't ever make it to LTS channels).

$ openssl version
OpenSSL 1.0.2g 1 Mar 2016

That means we have two options:

  1. Update OpenSSL manually and hope nothing system-critical actually needed the older version.
  2. Pick the best from what we've got for now.

The first involves a ton of extra work (it's all fairly straightforward, just super involved), so I'm going to cover that in a later post, once I'm finished with the current slate. That leaves us with making due with what we've got. Mozilla recommends prime256v1, secp384r1, and secp521r1 for modern compatibility (actually for any compatibility).

$ openssl ecparam -list_curves | grep -E "prime256v1|secp384r1|secp521r1"
secp384r1 : NIST/SECG curve over a 384 bit prime field
secp521r1 : NIST/SECG curve over a 521 bit prime field
prime256v1: X9.62/SECG curve over a 256 bit prime field

Nginx

1
ssl_ecdh_curve secp521r1:secp384r1:prime256v1;

Apache

1
SSLOpenSSLConfCmd Curves secp521r1:secp384r1:prime256v1

Generate Diffie-Hellman Group

Vanilla OpenSSL is susceptible to Logjam (among other things), so you'll want to create a new Diffie-Hellman group. Qualys mentions this as well; basically, don't use defaults.

Nginx

To generate,

$ sudo mkdir -p /etc/nginx/tls
$ sudo openssl dhparam -out /etc/nginx/tls/dhparams.pem 2048
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
.+.+.+...........

To use,

1
ssl_dhparam /etc/nginx/tls/dhparams.pem;

Apache

To generate,

$ sudo mkdir -p /etc/httpd/tls
$ sudo openssl dhparam -out /etc/httpd/tls/dhparams.pem 2048
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
.+.+.+...........

To use,

1
SSLOpenSSLConfCmd DHParameter "/etc/httpd/tls/dhparams.pem"

Use Server Cipher Preference

Having done all of this work to actually set up cipher precedence and curves and lots of other things, it's important to actually specify we'd prefer it if clients would use our configuration instead of theirs. Assuming all methods are the same (which is actually a horrible assumption), we have more control with our configuration.

Nginx

1
ssl_prefer_server_ciphers on;

Apache

1
SSLHonorCipherOrder on

OCSP Stapling

OCSP Stapling makes things a little bit simpler for Let's Encrypt. To verify a cert's veracity, clients historically had to send a request to the CA, negotiate that, and then, knowing the cert was valid, hit the intended address. OCSP stapling allows your server to periodically timestamp its validity through the magic of digital signatures.

Nginx

This config requires also setting ssl_trusted_certificate. This will be handled later, once we actually request a cert.

1
2
ssl_stapling on;
ssl_stapling_verify on;

It's fairly common to see a resolver defined as well, and equally common to see it defined as Google.

1
resolver 8.8.8.8 8.8.4.4;

I'm not a huge fan of this because it routes everything to Google. If left out, the resolver defaults to your DNS. Theoretically, my DNS already knows a user is checking me out. Google doesn't need to be involved for the same reasons OCSP stapling was created.

Apache

Apache doesn't make it easy. Here's a Stack Exchange thread that seems to cover the important stuff. This is the first of many times the new Apache cache will pop up, and, every single time, it requires manual setup.

1
2
SSLUseStapling On
SSLStaplingCache "shmcb:ssl_stapling(32768)"

SSL Session

As a fullstack dev who spends most of his time performing brute force data analysis and putting out fires, I honestly don't have a good baseline for what is or is not a safe config. Mozilla might have abandoned open communication to court the favor of big corporations, but they still occasionally support the little dev (just not the end user). The Mozilla TLS Config Generator is a fantastic tool to generate actually strong defaults. After reading several hundred posts about Nginx and Apache hardening for a couple of weekends, I've come to recognize the Mozilla standards pretty well. That was my long-winded way of saying this is section is total copypasta.

Due to a security issue, Mozilla doesn't recommend using session tickets. More recently, at least one named vulnerability has popped up regarding session tickets, so use them at your own risk.

Nginx

1
2
3
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;

Apache

Apache again makes this difficult. It does look like disabling tickets is easy, so they've got that going for them, which is nice.

1
2
SSLSessionCache "shmcb:ssl_scache(512000)"
SSLOpenSSLConfCmd Options -SessionTicket

Primary Config File Redux

Nginx

/etc/nginx/common/ssl.conf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
ssl_protocols TLSv1.2;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256;
ssl_ecdh_curve secp521r1:secp384r1:prime256v1;
ssl_dhparam /etc/nginx/tls/dhparams.pem;
ssl_prefer_server_ciphers on;

ssl_stapling on;
ssl_stapling_verify on;

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;

Apache

For the nth time, I'd like to reiterate that I haven't actually tested this config. I will. Eventually.

/etc/httpd/common/ssl.conf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
SSLStaplingCache "shmcb:/path/to/ssl_stapling(32768)"
SSLSessionCache "shmcb:/path/to/ssl_scache(512000)"

<VirtualHost *:443>
SSLProtocol -all +TLSv1.2
SSLCipherSuite "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256"
SSLOpenSSLConfCmd Curves secp521r1:secp384r1:prime256v1
SSLOpenSSLConfCmd DHParameter "/etc/httpd/tls/dhparams.pem"
SSLHonorCipherOrder on
SSLUseStapling On
SSLOpenSSLConfCmd Options -SessionTicket
</VirtualHost>

Before You Go

Let's Encrypt is a fantastic service. If you like what they do, i.e. appreciate how accessible they've made secure web traffic, please donate. EFF's certbot is what powers my site (and basically anything I work on these days); consider buying them a beer (it's really just a donate link but you catch my drift).

Legal Stuff

I'm still pretty new to the whole CYA legal thing. I really like everything I've covered here, and I've done my best to respect individual legal policies. If I screwed something up, please send me an email ASAP so I can fix it.

  • The Electronic Frontier Foundation and certbot are covered by EFF's generous copyright. As far as I know, it's all under CC BY 3.0 US. I made a few minor tweaks to build the banner image but tried to respect the trademark. I don't know who the certbot logo artist is but I really wish I did because it's a fantastic piece of art.
  • Let's Encrypt is trademarked. Its logo uses CC BY-NC 4.0. I made a few minor tweaks to build the banner image but tried to respect the trademark.
  • I didn't find anything definitive (other than EULAs) covering Nginx, which doesn't mean it doesn't exist. Assets were taken from its press page.
  • Apache content was sourced from its press page. It provides a full trademark policy.

CJ Harries

I did a thing once. Change "blog." to "cj@" and you've got my email. All these opinions are mine and might not be shared by clients or employers.

Read More