How To Enable HTTPS for Varnish Cache using Hitch on CentOS-RHEL 8 - DesignLinux

May 29 2020
designlinux 0 Comments

Varnish Cache lacks native support for SSL/TLS and other protocols associated with port 443. If you are using Varnish Cache to boost your web application’s performance, you need to install and configure another piece of software called an SSL/TLS termination proxy, to work alongside Varnish Cache to enable HTTPS.

The Hitch is a free open source, libev-based, and scalable SSL/TLS proxy designed for Varnish Cache, which currently works on Linux, OpenBSD, FreeBSD, and MacOSX. It terminates TLS/SSL connections by listening on port 443 (the default port for HTTPS connections) and forwards the unencrypted traffic to Varnish Cache, however, it should work with other backends too.

It supports for TLS1.2 and TLS1.3 and legacy TLS 1.0/1.1, supports ALPN (Application-Layer Protocol Negotiation) and NPN (Next Protocol Negotiation) for HTTP/2, a PROXY protocol to signal client IP/port to a backend, UNIX domain socket connections to the origin, SNI (Server Name Indication), with and without wildcard certificates. Additionally, it works well for large installations that require up to 15,000 listening sockets and 500,000 certificates.

As a continuation of our two previous articles about installing Varnish Cache for Nginx and Apache HTTP servers, this guide shows to enable HTTPS for Varnish Cache using Hitch TLS Proxy on CentOS/RHEL 8.

Logical Diagram of Our Setup

Logical Diagram of Our Setup

This guide assumes that you have installed Varnish for Nginx or Apache web server, otherwise, see:

Step 1: Install Hitch on CentOS/RHEL 8

1. The Hitch package is provided in the EPEL (Extra Packages for Enterprise Linux) repository. To install it, first enable EPEL on your system and then install the package thereafter. If you do not have OpenSSL package installed, install it as well.

# dnf install epel-release
# dnf install hitch openssl
Install Hitch on CentOS 8

Install Hitch on CentOS 8

2. When the package installation is complete, you will have to configure Varnish Cache to work Hitch. You also need to configure Hitch to use your SSL/TLS certificates and Varnish as a backend. The main configuration file of Hitch is located at /etc/hitch/hitch.conf, which is explained below.

Step 2: Configuring Varnish Cache for Hitch

3. Next, enable Varnish to listen to an additional port (8443 in our case) using the PROXY protocol support, for communications with Hitch.

So open the Varnish systemd service file for editing.

# systemctl edit --full varnish

Look for the line ExecStart and add an additional -a flag with the value 127.0.0.1:8443,proxy. Using a value of 127.0.0.1:8443 means Varnish will only accept the internal connection (from processes running on the same server i.e hitch in this case) but not external connections.

ExecStart=/usr/sbin/varnishd -a :80 -a 127.0.0.1:8443,proxy -f /etc/varnish/default.vcl -s malloc,256m
Set Varnish Listen Port for Hitch

Set Varnish Listen Port for Hitch

Save the file and then restart the Varnish service to apply the latest changes.

# systemctl restart varnish

Step 3: Obtaining SSL/TLS Certificates

4. In this section, we will explain how to create the SSL/TLS certificate bundle to be used under Hitch. For this guide, we will explain the different options of how to use a self-signed certificate, commercial certificate, or one from Let’s Encrypt.

To create a self-signed certificate (which you should only use in a local testing environment), you can use the OpenSSL tool.

# mkdir /etc/ssl/tecmint.lan
# cd /etc/ssl/tecmint.lan/
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout  tecmint.lan.key -out tecmint.lan.crt

Then create a bundle of the certificate and key as follows.

# cat tecmint.crt tecmint.key >tecmint.pem

Note: For production use, you can either buy a certificate from a commercial Certificate Authority (CA) or grab a free, automated, and fully recognized certificate from Let’s Encrypt. Then create a PEM bundle.

If you bought a certificate from a commercial CA, you need to merge the private key, the certificate, and the CA bundle as shown.

# cat example.com.key example.com.crt example.com-ca-bundle.crt > /etc/ssl/example.com.pem

For Let’s Encrypt, the certificate, private key, and the full chain will be stored under /etc/letsencrypt/live/example.com/, so create the bundle as shown.

# cat /etc/letsencrypt/live/example.com/fullchain.pem /etc/letsencrypt/live/example.com/privkey.pem >/etc/letsencrypt/live/example.com/example.com_bundle.pem

Step 4: Configuring and Starting Hitch

5. Next, configure Varnish as a backend for Hitch and specify the SSL/TLS certificate files to use for HTTPS, in the Hitch main configuration file, open it for editing.

# vi /etc/hitch/hitch.conf

The frontend section defines the IP addresses and port Hitch will listen to. The default configuration is to listen on all IPv4 and IPv6 interfaces attached on the server and runs on port 443 and handle incoming HTTPS requests, handing them off to Varnish.

Change the default backend proxy port from 6086 to 8443 (the port used to forward requests to Varnish) in the Hitch configuration file, using the backend parameter. Also, specify the certificate file using the pem-file parameter as shown.

backend = "[127.0.0.1]:8443"
#pem-dir = "/etc/pki/tls/private"
pem-file = "/etc/ssl/tecmint.lan/tecmint.pem"
Configure Hitch as SSL/TLS Proxy for Varnish

Configure Hitch as SSL/TLS Proxy for Varnish

Save the file and close it.

6. Now start the hitch service and enable it to automatically start at system boot. Note that the --now switch when used with enable, starts a systemd service as well and then check status to see if it is up and running as follows.

# systemctl enable --now hitch
# systemctl status hitch
Verify Hitch Status

Verify Hitch Status

7. Before you proceed to test if your web site/application is now running on HTTPS, you need to allow the HTTPS service port 443 in the firewall to allow requests destined for that port on the server to pass through the firewall.

# firewall-cmd --zone=public --permanent --add-service=https
# firewall-cmd --reload

Step 5: Testing SSL/TLS Termination with Varnish Cache-Hitch Setup

8. It’s now time to test the Varnish Cache-Hitch setup. Open a web browser and use your domain or server’s IP to navigate over HTTPS.

https://www.example.com
OR
https://SERVER_IP/

Once the index page of your web application has loaded, check the HTTP headers to confirm that content is being served via Varnish Cache.

To do that, right-click on the loaded web page, select Inspect from the list of options to open the developer tools. Then click on the Network tab, and Reload the page, then select a request to view the HTTP headers, as highlighted in the following screenshot.

Check HTTPS Setup on Varnish Cache

Check HTTPS Setup on Varnish Cache

Step 6: Redirecting HTTP to HTTPS in Varnish Cache

9. To run your web site on HTTPS only, you need to redirect all HTTP traffic to HTTPS. You can do this by adding the following configuration in your Hitch configuration file.

# vi /etc/hitch/hitch.conf

First, add the line import std; just below vlc 4.0;, then look for the vlc_recv subroutine, which is the first VCL subroutine executed immediately after Varnish Cache has parsed the client request into its basic data structure. It is where we can modify the request headers and execute a synth to redirect client requests.

Modify it to look like this.

sub vcl_recv {
    if (std.port(server.ip) != 443) {
        set req.http.location = "https://" + req.http.host + req.url;
        return(synth(301));
    }
}

Note that the PROXY protocol enables Varnish to see Hitch’s listening port 443 from the server.ip variable. So the line std.port(server.ip) returns the port number on which the client connection was received.

If the port is not 443 for HTTPS (as checked by (std.port(server.ip) != 443)), the subroutine will set the request HTTP Location header (set req.http.location) to a secure request (“https://” + req.http.host + req.url) simply asking the web browser to load a HTTPS version of the web page (i.e URL redirection).

The Location header will be sent to the vcl_synth subroutine (which is called using return(synth(301))) with an HTTP status code of 301 (Moved permanently).

10. Next, add the following vcl_synth subroutine (one of its many uses cases is redirecting users), to process the synth above.

sub vcl_synth {
        if (resp.status == 301) {
                set resp.http.location = req.http.location;
		  set resp.status = 301;
                return (deliver);
        }
}

It checks if the response status is 301, the HTTP Location header in the response is set to the HTTP Location header in the request which is in fact a redirect to HTTPS and executes a deliver action.

The deliver action builds a response with the response from the backend, stores the response in the cache, and sends it to the client.

Configure Hitch to Redirect HTTP to HTTPS

Configure Hitch to Redirect HTTP to HTTPS

Save the file and close it.

11. Once again, apply the new changes in the Varnish configuration by restarting the service. Then use the curl command-line tool to confirm redirection from HTTP to HTTPS.

# systemctl restart varnish
# curl -I http://eaxmple.com/
Check HTTP to HTTPS Redirect

Check HTTP to HTTPS Redirect

From the browser, the response is also the same as shown in the following screenshot.

Check HTTP to HTTPS Redirection in Browser

Check HTTP to HTTPS Redirection in Browser

We hope that everything has worked just fine up to this point. If not, drop a comment or questions via the feedback form below. For any advanced configuration options, go to the Varnish Cache documentation and Hitch documentation.

Tags: , ,