HAProxy and Let’s Encrypt

Today we’re going to talk about reverse proxy with fully automated SSL certificate handling. We’re going to take a look into HAProxy and Let’s Encrypt in conjunction.

First some relevant background information. Why would you want a reverse proxy? There could be numerous reasons like

• Load balancing a service between multiple servers
• Do SSL offloading (i.e. handle all certificates and encryption on one server only)
• Publish multiple services with the same port number on a single IP

For my home environment, I need a reverse proxy mainly for publishing multiple services using the same port on a single external IP. To achive that, I have implemented HAProxy to look at the header and redirect the traffic based on that. Since all traffic is passing throught HAProxy, I decided to handle all my certificates there as well.

The solution will look something like this:

Configure HAProxy

I won’t cover all the details on how to install HAProxy. There are plenty of great guides out there in how to install and configure the basics.

Frontends

This is what the clients will communicate with. We’ll have to configure one frontend for each port we want to publish. I have configured two frontends, one for HTTP and one for HTTPS.

HTTP

This frontend listens for connections on port 80. I have configured it to redirect all incoming requests to HTTPS.

# HTTP Frontend
frontend http-in
    bind *:80

    # Redirect to HTTPS
    redirect scheme https

HTTPS

This frontend listens for connections on port 443. It will also handle the SSL encryption. Here, I have setup a set of ACLs to redirect the traffic to the correct backend server based on the header sent by the client. I.e. if a client brows to https://myapp1.domain.com/index.html, HAProxy will read that address and redirect the traffic to webserver1. I have also configured a fallback destination to use if no ACL applies.

# HTTPS Frontend
frontend https-in
    bind *:443 ssl crt /etc/ssl/mycertificate.pem
        
     # ACLs
    acl acl_myapp1    hdr_end(host) -i myapp1.domain.com
    acl acl_myapp2    hdr_end(host) -i myapp2.domain.com
    acl acl_mail      hdr_end(host) -i mail.domain.com

    # Rules
    use_backend webserver1_http if acl_myapp1
    use_backend webserver2_http if acl_myapp2
    use_backend mailserver1_http if acl_mail

    # Fallback rule
    default_backend webserver1_http

Backends

Now that we are having our frontends setup, HAProxy can communicate with a client and it also knows what to look for to determine the end destination. Now we need to configure these destinations, called backends.

# Backends
backend webserver1_http
    server webserver1.domain.com 192.168.1.1:80

backend webserver2_http
    server webserver2.domain.com 192.168.1.2:80

backend mailserver1_http
    server mailserver1.domain.com 192.168.1.3:80

Now we should have a working configuration, except for the SSL part.

Configure Let’s Encrypt

To start of with, we must install Certbot. Certbot will be used to communicate with Let’s Encrypt using ACME. On an Ubuntu server, run the following command:

apt install certbot

Now, here’s the problem. When trying to issue a standard certificate (not a wildcard certificate), the Certbot will be instructed to place a file in the requested domain’s web server root in order for Let’s Encrypt to validate that you actually own the domain in question. When Let’s Encrypt trying to read the file, HAProxy will treat the traffic as any client and redirect it to a backend – where ther is neither Certbot nor a validation file.

We somehow need to tell HAProxy that ACME traffic must remain in HAProxy. We also need to instruct Certbot where to place the validation file. Luckily, this guy has built a plugin to HAProxy that handles it.

Install ACME Validation Plugin

Download the plugin, extract it and place it in HAproxy’s configuration folder. On an Ubuntu server, run the following commands:

cd
wget https://github.com/janeczku/haproxy-acme-validation-plugin/archive/master.zip
unzip master.zip
mv master/acme-http01-webroot.lua /etc/haproxy
mv master/cert-renewal-haproxy.sh /etc/haproxy

The bash script will handle the certification issuing process by talking to Let’s Encrypt, placing the validation file in the correct folder, reload HAProxy, etc. The LUA script will then handle the ACME traffic and make sure to serve the validation file to Let’s Encrypt (instead of redirecting the traffic to a backend server).

Put the following in haproxy.cfg to start using the LUA script:

global
    ...
    lua-load    /etc/haproxy/acme-http01-webroot.lua
 

# HTTP Frontend
frontend http-in
    ...
    # ACME Handler for Lets Encrypt
    acl url_acme_http01 path_beg /.well-known/acme-challenge/
    http-request use-service lua.acme-http01 if METH_GET url_acme_http01

Now, reload HAProxy.

systemctl reload haproxy

So far so good! Now we should be able to issue a certificate, but don’t do it yet! We need to alter the bash script a bit.

First you need to understand how Certbot and HAProxy works. When issuing a certificate, Certbot will place it in a folder specific for that domain. It will also keep the certificate and the private key apart. HAProxy on the other hand needs to have all the parts in one single PEM file. It also need to know the full path to the file.

Of course, the author of the script thought about this and implemented a solution that merges the certificate and the key into a single PEM file. The issue with this is that those PEM files will still be located in separate folders specific to each domain. Practically this means that you have to tell HAproxy about each and every of your PEM files. This is not a problem if you only have one domain and one certificate, but most probably that’s not the case.

Luckily, HAProxy can include a whole folder with PEM files, meaning that you can add or remove certificates on the fly. HAProxy will use SNI to determine what certificate to serve to the client based on the requested domain name.

For this to work, we need to tell the bash script to place the merged PEM file in a common folder. In cert-renewal-haproxy.sh, replace the line

cat ${le_cert_root}/${domain}/privkey.pem ${le_cert_root}/${domain}/fullchain.pem | tee ${le_cert_root}/${domain}/haproxy.pem >/dev/null

with

cat ${le_cert_root}/${domain}/privkey.pem ${le_cert_root}/${domain}/fullchain.pem | tee ${le_cert_root}/pem/${domain}.pem >/dev/null

Now, when a certificate is renewed, the merged PEM file will be stored as www.domain.com.pem in a folder called pem, located in Certbot’s live folder (e.g. /etc/letsencrypt/live). Before we continue, we need to create this folder.

mkdir /etc/letsencrypt/live/pem

Install Certificate

Now it’s finally time to install a certificate! Run the following commands to issue a certificate and create the PEM file for HAProxy:

certbot certonly --text --webroot --webroot-path /var/lib/haproxy -d www.domain.com --renew-by-default --agree-tos --email administrator@domain.com
cat /etc/letsencrypt/live/www.domain.com/privkey.pem /etc/letsencrypt/live/www.domain.com/fullchain.pem > /etc/letsencrypt/live/pem/www.domain.com.pem

Re-run the commands at any time to issue a new certificate.

Now when we have at least one certifiate in our pem folder, we need to configure HAProxy to use these certificate(s).

Change the following in haproxy.cfg:

# HTTPS Frontend
frontend https-in
        bind *:443 ssl crt /etc/letsencrypt/live/pem/

And reload HApxory (don’t forget to reload HAProxy each time you issue a new certificate).

systemctl reload haproxy

Now give it a try! You should reach your backend service as well as being presented with a valid certificate.

Configure Automatic Renewal

Now when we’re have everything up and running, there’s one step left – the automatic renewal process.

When installing Certbot, a cron task is automatically beeing added to the system to handle certificate renewals. This will only renew certifcates and not create any PEM files, nor will it reload HAProxy. For this, we need to alter the cron task to run our awesome bash script instead.

In /etc/cron.d/certbot, replace the line

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

with

0 */12 * * * root /etc/haproxy/cert-renewal-haproxy.sh

That’s it! We now have a reverse proxy in place that automatically handles everything around SSL as well as publishing various services on the same frontend.