OpenBSD, Let’s Encrypt, and Full Certificate Chains

When I setup zacbrown.org, I used Let’s Encrypt to setup my SSL certificate. I’m using OpenBSD to host the site and it provides acme-client as the built-in tool to request a Let’s Encrypt certificate.

So as part of setting up acme-client, I setup the following in /etc/acme-client.conf:

domain www.zacbrown.org {
	alternative names { zacbrown.org www.zacbrown.net zacbrown.net www.zacbrown.dev zacbrown.dev }
	domain key "/etc/ssl/private/www.zacbrown.org.key"
	domain certificate "/etc/ssl/www.zacbrown.org.crt"
	domain full chain certificate "/etc/ssl/www.zacbrown.org.pem"
	sign with letsencrypt
}

I then used a coupole of sites as references for setting up relayd, httpd, and acme-client for my site site:

Following the instructions verbatim works great except for one issue. I noticed when I asked SSLLab’s SSL Server Test that it was telling me that my certificate chain was incomplete. That is, it was missing a full chain:

I found that a little confusing given that I specified the domain full chain certificate ... line in /etc/acme-client.conf. I expected that relayd would have picked that file up as part of its configuration for TLS termination.

Let’s read the manpage for relayd about the tls keypair:

The relay will attempt to look up a private key in /etc/ssl/private/name:port.key and a public certificate in /etc/ssl/name:port.crt, where port is the specified port that the relay listens on. If these files are not present, the relay will continue to look in /etc/ssl/private/name.key and /etc/ssl/name.crt. This option can be specified multiple times for TLS Server Name Indication. If not specified, a keypair will be loaded using the specified IP address of the relay as name. See ssl(8) for details about SSL/TLS server certificates.

Oh - hmmm. So it’s looking specifically at the file ending in .crt. We know that the .pem file contains the full chain but what does the .crt file have in it? It contains just the certificate content for www.zacbrown.org but nothing about the rest of the chain:

root@host /e/ssl# diff www.zacbrown.org.crt www.zacbrown.org.pem
37a38,64
>
> -----BEGIN CERTIFICATE-----
> MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
> MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
> DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
> MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
> AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
> jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
> Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
> U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
> gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
> /xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
> oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
> BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
> ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
> p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
> AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
> Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
> LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
> r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
> AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
> ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
> S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
> qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
> O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
> UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
> -----END CERTIFICATE-----

Ah - so the diff indicates there’s one additional certificate in the .pem file that isn’t in the .crt file.

Let’s print some info about the contents of the .crt file:

root@zacbrown /e/ssl# openssl crl2pkcs7 -nocrl -certfile www.zacbrown.org.crt | openssl pkcs7 -print_certs -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:39:ff:2a:e1:59:56:ee:71:98:ae:0a:b6:a7:fd:4c:22:17
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=R3
        Validity
            Not Before: Mar 14 21:51:41 2021 GMT
            Not After : Jun 12 21:51:41 2021 GMT
        Subject: CN=www.zacbrown.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    ...<TRUNCATED>...
                Exponent: 65537 (0x10001)
        ...<TRUNCATED>...

Alright - so like the manpages told us, the .crt file contains the leaf certificate for www.zacbrown.org.

Let’s print some info about each of the certificates in the .pem file and and see how that compares:

root@zacbrown /e/ssl# openssl crl2pkcs7 -nocrl -certfile www.zacbrown.org.pem | openssl pkcs7 -print_certs -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:39:ff:2a:e1:59:56:ee:71:98:ae:0a:b6:a7:fd:4c:22:17
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=R3
        Validity
            Not Before: Mar 14 21:51:41 2021 GMT
            Not After : Jun 12 21:51:41 2021 GMT
        Subject: CN=www.zacbrown.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    ...<TRUNCATED>...
                Exponent: 65537 (0x10001)
    ...<TRUNCATED>...

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:01:75:04:83:14:a4:c8:21:8c:84:a9:0c:16:cd:df
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Oct  7 19:21:40 2020 GMT
            Not After : Sep 29 19:21:40 2021 GMT
        Subject: C=US, O=Let's Encrypt, CN=R3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
    ...<TRUNCATED>...

Aha! So the second certificate in the .pem file is the intermediate certificate from Let’s Encrypt that issued the certificate for www.zacbrown.org.

The root certificate, named DST Root CA X3, is likely to already be in most browser’s trust stores so as long as we present a full chain rooted through that we should be good.

Wait, so what to do to fix our intermediate chain?

Well - what if we just copy the contents of www.zacbrown.org.pem into www.zacbrown.org.crt. We know from the relayd.conf manpage that it will use www.zacbrown.org.crt by default. So if that file contains the full chain then maybe it will just work?

(I know this was a little bit of a leap - I was mostly just experimenting and surprised to find out it worked.)

So in effect, I made a backup of www.zacbrown.org.crt and copied www.zacbrown.org.pem to www.zacbrown.crt:

root@zacbrown /e/ssl# mv www.zacbrown.org.crt www.zacbrown.org.crt-bak
root@zacbrown /e/ssl# cp www.zacbrown.org.pem www.zacbrown.org.crt
root@zacbrown /e/ssl# rcctl restart relayd
relayd(ok)
relayd(ok)

Now, when we run the SSLLabs scan tool we got a happy A+ report:

But wait, there’s even more!

Copying the contents of the .pem file into the .crt file is well and good, except when it comes time to renew the Let’s Encrypt cert in 6 months it’ll get overwritten. What to do?

Easy - we just modify the /etc/acme-client.conf file to rename the target files when it writes the certificate material:

~/C/s/s/z/etc ❯❯❯ git diff
diff --git a/zacbrown.org/etc/acme-client.conf b/zacbrown.org/etc/acme-client.conf
index fe39f07..74bbb12 100644
--- a/zacbrown.org/etc/acme-client.conf
+++ b/zacbrown.org/etc/acme-client.conf
@@ -26,7 +26,7 @@ authority buypass-test {
 domain www.zacbrown.org {
        alternative names { zacbrown.org www.zacbrown.net zacbrown.net www.zacbrown.dev zacbrown.dev }
        domain key "/etc/ssl/private/www.zacbrown.org.key"
-       domain certificate "/etc/ssl/www.zacbrown.org.crt"
-       domain full chain certificate "/etc/ssl/www.zacbrown.org.pem"
+       domain certificate "/etc/ssl/www.zacbrown.org.crt.leaf"
+       domain full chain certificate "/etc/ssl/www.zacbrown.org.crt"
        sign with letsencrypt
 }

Now we’ll create what was our .crt file as a .crt.leaf file and we’ll put the contents of what was the .pem file into the .crt file. relayd and httpd will be none the wiser when our cronjob for acme-client rolls the certificate.



Posted on 2021-04-10