Install HAProxy & CertBot (LetsEncrypt)

HAProxy

Install HAProxy

Install HAProxy v2.9 with the following commands:

sudo add-apt-repository -y ppa:vbernat/haproxy-2.9
sudo apt update
sudo apt install -y haproxy

Install WhoIs for Basic Password Protection of Statistics Page (Optional)

Install WhoIs (which includes the mkpasswd utility) with the following command:

sudo apt install whois

Generate an encrypted password for the config file for accessing the statistics with the following command:

echo <password> | mkpasswd --stdin --method=sha-256

Configure HAProxy

Edit the HAProxy configuration with this command:

sudo nano /etc/haproxy/haproxy.cfg

Inside the global section, add the following lines to the end of the section to allow easier loading of certificates and to force stopping the old process on a reload (which will drop all long-standing connections):

	ssl-load-extra-files all
	ssl-load-extra-del-ext
	
	# This forces open connections to close after reloading HAProxy - Use with caution in production
	hard-stop-after 30s

Leave the defaults section as they are but add a frontend for stats and normal web access:

userlist StatsUsers
	user <username> password <salted_password>

frontend http_fe
	bind *:80
	bind *:443 ssl crt /etc/ssl/private

    # Set some default forwarded headers
	option forwardfor
	http-request set-header X-Forwarded-Port %[dst_port]
	http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
	http-request set-header X-Forwarded-Proto https if { ssl_fc }
	http-request add-header X-Forwarded-Host %[req.hdr(Host)]

	# Capture additional/longer info for logs
	http-request capture req.hdr(Host) len 30
	capture request header User-Agent len 200
	capture request header Referer len 800
	capture request header X-Forwarded-For len 20

	# Test URI to see if it's a LetsEncrypt request
	acl letsencrypt_acl path_beg /.well-known/acme-challenge/

	# Local Stats Authorization
	acl stats_acl path_beg /hastats
	acl stats_auth http_auth(StatsUsers)
	http-request auth realm Stats if stats_acl !stats_auth

	# Common Redirects
	acl <redirect_acl_name> hdr_beg(host) -i <domain_prefix>
	http-request redirect location <url> if <redirect_acl_name>

	# The following redirect for non-https traffic breaks if used with Cloudflare Flexible Encryption
	redirect scheme https code 301 if !{ ssl_fc }

	# Local Stats requests
    use_backend stats_be if stats_acl stats_auth

	# LetsEncrypt Requests
    use_backend certbot_be if letsencrypt_acl

	# <Name of website or any comment - Create these lines for every domain/backend>
	acl <acl_name> hdr_beg(host) -i <domain_prefix>.
	use_backend <backend_name> if <acl_name>

	# Default backend if no ACL rules are matched (Can be deleted for no default)
	default_backend <default_backend_name>

# <Name of the website or any comment - Create this section for every domain/backend>
backend <backend_name>
	balance roundrobin

    option httpchk
    http-check send meth HEAD uri /

	default-server check maxconn 20
	server <server_name> <server_ip>:<server_port>

# Local stats
backend stats_be
	stats enable
	stats uri /hastats
	stats refresh 10s
	stats show-modules

# Certbot for LetsEncrypt
backend certbot_be
	server certbot 127.0.0.1:8888

Here's an example using an Ignition Server running on 192.168.1.10 using a domain name beginning with scada. for remote access. This also enables HAProxy stats on any domain name using the path /hastats using a login of hastats and a password of password:

userlist StatsUsers
	user hastats password $5$cQEGcl6Wb$nG3i246htOnuW5LTYo3O73aa7ve7i.ePiQ4v6EoMW44

frontend http_fe
	bind *:80
	bind *:443 ssl crt /etc/ssl/private

    # Set some default forwarded headers
	option forwardfor
	http-request set-header X-Forwarded-Port %[dst_port]
	http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
	http-request set-header X-Forwarded-Proto https if { ssl_fc }
	http-request add-header X-Forwarded-Host %[req.hdr(Host)]

    # Capture additional/longer info for logs
	http-request capture req.hdr(Host) len 30
	capture request header User-Agent len 200
	capture request header Referer len 800
	capture request header X-Forwarded-For len 20

	# Test URI to see if it's a LetsEncrypt request
	acl letsencrypt_acl path_beg /.well-known/acme-challenge/

	# Local Stats Authrization
	acl stats_acl path_beg /hastats
	acl stats_auth http_auth(StatsUsers)
	http-request auth realm Stats if stats_acl !stats_auth

	# Ignition Perspective Redirect to Project URL
	acl scada hdr_beg(host) -i <redirected-subdomain>.
	http-request redirect location https://<subdomain>.<domain>.com/data/perspective/client/<ProjectName> if scada

	# The following redirect for non-https traffic breaks if used with Cloudflare Flexible Encryption
	redirect scheme https code 301 if !{ ssl_fc }
    
	# Local Stats requests
	use_backend stats_be if stats_acl stats_auth

	# LetsEncrypt requests
	use_backend certbot_be if letsencrypt_acl

	# Ignition
	acl ignition hdr_beg(host) -i <subdomain>.
	use_backend ignition_be if ignition

# Ignition
backend ignition_be
	balance roundrobin

    option httpchk
    http-check send meth HEAD uri / ver HTTP/1.1 hdr host 127.0.0.1

	http-request set-header Connection "Upgrade"

	default-server check maxconn 1000
	server iguana 192.168.1.10:8088 check inter 10s

# Local stats
backend stats_be
	stats enable
	stats uri /hastats
	stats refresh 10s
	stats show-modules

# Certbot for LetsEncrypt
backend certbot_be
	server certbot 127.0.0.1:8888

For a very simple HAProxy in front of a local Ignition Edge server, use the following config:

userlist StatsUsers
	user hastats password $5$cQEGcl6Wb$nG3i246htOnuW5LTYo3O73aa7ve7i.ePiQ4v6EoMW44

frontend http_fe
	bind *:80

    # Set some default forwarded headers
	option forwardfor
	http-request set-header X-Forwarded-Port %[dst_port]
	http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
	http-request set-header X-Forwarded-Proto https if { ssl_fc }
	http-request add-header X-Forwarded-Host %[req.hdr(Host)]

	# Capture additional/longer info for logs
	http-request capture req.hdr(Host) len 30
	capture request header User-Agent len 200
	capture request header Referer len 800
	capture request header X-Forwarded-For len 20

	# Local Stats Authrization
	acl stats_acl path_beg /hastats
	acl stats_auth http_auth(StatsUsers)
	http-request auth realm Stats if stats_acl !stats_auth

	# Local Stats requests
	use_backend stats_be if stats_acl stats_auth

	# Ignition
	use_backend ignition_be

# Ignition
backend ignition_be
	balance roundrobin

    option httpchk
    http-check send meth HEAD uri / ver HTTP/1.1 hdr host 127.0.0.1

	http-request set-header Connection "Upgrade"

	default-server check maxconn 1000
	server ignition 127.0.0.1:8088 check inter 10s

# Local stats
backend stats_be
	stats enable
	stats uri /hastats
	stats refresh 10s
	stats show-modules

Any time you modify the configuration for HAProxy, you can check the configuration for errors with the following command:

sudo haproxy -f /etc/haproxy/haproxy.cfg -c

If the configuration file is valid (warnings are OK) you can reload HAProxy with the following command:

sudo service haproxy reload

Upgrade HAProxy

If you ever need to upgrade the version to a newer version, check the following site to see if the newer version exists:

https://launchpad.net/~vbernat/+ppa-packages

If it does, use the above commands substituting in the appropriate version to add the newer version's repository and upgrade HAProxy. Once HAProxy is upgraded and verified working, use the following command to remove the old version (again substituting the old version number):

sudo add-apt-repository --remove ppa:vbernat/haproxy-2.8

Certbot (LetsEncrypt)

Install CertBot (LetsEncrypt)

Install Certbot with the following commands:

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo snap set certbot trust-plugin-with-root=ok

Create a new certificate using HTTP authentication

We'll need to get a new certificate for the domain name we want using the following command:

sudo certbot certonly --standalone -d <domain_name> --non-interactive --agree-tos --email <admin_email> --http-01-port=8888 --preferred-chain "ISRG Root X1"

If this fails, you'll need to comment out the following 2 lines in your HAProxy configuration so that it only listens on port 80 and not on port 443:

#	bind *:443 ssl crt /etc/ssl/private
#	redirect scheme https code 301 if !{ ssl_fc }

Don't forget to reload HAProxy after making this change and once you've created the certificate, you can un-comment these lines and reload HAProxy again.

Create a new certificate using DNS authentication

A list of DNS services that are supported along with the plugin names can be found here: https://certbot.eff.org/docs/using.html#dns-plugins

We'll need to install a plugin for the DNS service we will be using with the following command:

sudo snap install certbot-dns-<plugin>

Next you'll create your credentials file using the following command and putting the credentials needed as documented in the plugin documentation link from above:

sudo nano /root/.secrets/certbot/<dns_service_name>.ini

Next you'll make these credentials read-only for the root user so that any other user cannot see them:

sudo chmod 0400 /root/.secrets/certbot/<dns_service_name>.ini

Now we'll get our new certificate for the domain name we want using the following command (adjust as necessary according to the plugin documentation in the link above):

sudo certbot certonly --non-interactive --agree-tos --email <admin_email> --dns-<dns_service_name> --dns-<dns_service_name>-credentials /root/.secrets/certbot/<dns_service_name>.ini -d <domain> --preferred-chain "ISRG Root X1"

If wanting a wildcard certificate, you'll need to use the following command instead (adjust as necessary according to the plugin documentation in the link above):

sudo certbot certonly --non-interactive --agree-tos --email <admin_email> --dns-<dns_service_name> --dns-<dns_service_name>-credentials /root/.secrets/certbot/<dns_service_name>.ini -d <domain> -d *.<domain> --preferred-chain "ISRG Root X1"

Preparing the certificate for HAProxy

To prepare the certificate in a format HAProxy can use, we'll create symbolic links to the 2 files using the following commands:

sudo ln -s /etc/letsencrypt/live/<domain>/fullchain.pem /etc/ssl/private/<domain>.crt

sudo ln -s /etc/letsencrypt/live/<domain>/privkey.pem /etc/ssl/private/<domain>.key

Schedule auto-renewal of certificates

We'll create a script which we'll run on a schedule to renew the certificate:

sudo nano /opt/update-certs.sh

Put the following text (update as needed) in the file:

#!/usr/bin/env bash
# Renew the certificate using TLS authentication on prot 8888 (comment out this line if using DNS authentication)
certbot renew --force-renewal --https-port=8888 --preferred-chain "ISRG Root X1"

# Renew the certificate using DNS authentication (uncomment if using DNS authentication)
#certbot renew --force-renewal --preferred-chain "ISRG Root X1"

# Reload  HAProxy
service haproxy reload

We'll edit the certbot cron scheduler file to schedule this script:

sudo nano /etc/cron.d/certbot

This schedule will be used to run our renewal script on the 1st day of the month every month at midnight (add it to a new line of the file and comment out the existing schedule using a # sign):

0 0 1 * * root bash /opt/update-certs.sh

Removing or Cancelling Renewing of Certificates

If you ever need to stop renewing a certificate that is no longer used, you can do so with the following command:

sudo certbot delete --cert-name <domain>