Reverse proxy and TLS/SSL termination options

TLS enables HTTPS to encrypt the message between two parties, but the message needs to be decrypted in the end so that each party can read it. In Jenkins, there are two places where the message can be decrypted, or in industry lingo, where the TLS is terminated.

TLS termination at the reverse proxy

A popular configuration is to set up a reverse proxy in front of Jenkins and terminate the TLS at the reverse proxy:

• DNS resolves the URL to the IP of the reverse proxy.
• If the traffic is on HTTP port 80, the reverse proxy redirects the traffic to HTTPS port 443.
• If the traffic is on HTTPS port 443, the reverse proxy decrypts the message and forwards it to the Jenkins controller on HTTP port 8080.
• The Jenkins controller processes the incoming message from HTTP port 8080.

This is what the configuration looks like:

Figure 2.7 – Architecture of Jenkins with reverse proxy

There are several benefits to this configuration:

• The reverse proxy acts as a public face of the application. Therefore, the applications can stay in a private network where an attacker cannot directly access.
• Attacks are easier to mitigate since reverse proxies are built to handle attacks. In AWS, Elastic Load Balancer (ELB) can act as a reverse proxy and provide additional protections through AWS Shield.
• Traffic to HTTP is automatically redirected to HTTPS.
• Connections are pooled and compressed for higher performance.
• The processing cost of terminating the TLS is offloaded to the reverse proxy, however small it is in reality.

There are also drawbacks:

• It adds to the amount of infrastructure to manage.
• Communication between the reverse proxy and Jenkins can fail due to misconfigurations. Reverse proxy issues are, in fact, quite common.
• Communication between the reverse proxy and Jenkins is unencrypted, creating a window of opportunity for an attack, however small it is in reality.

Let's examine the steps for configuring a reverse proxy for each Jenkins controller.

Setting up an Application ELB for the AWS Jenkins controller

Setting up an ELB for Jenkins is largely a three-part exercise:

1. Create a TLS certificate through AWS Certificate Manager.
2. Create a target group that does the following:
   • Has the Jenkins controller EC2 instance
   • Listens to port 8080
   • Runs health checks on the /login path
3. Create a load balancer that does the following:
   • Listens to port 80, which is redirected to port 443
   • Listens to port 443, which is forwarded to the target group from step 2
   • Uses the TLS certificate from step 1

Here are the actual steps that we can follow:

1. From the AWS console, go to the EC2 service and click Load Balancers.
2. Create a new Application Load Balancer named jenkins. Add two listeners, HTTP and HTTPS. Choose the availability zones for the ELB and click Next.
3. Create a certificate in AWS Certificate Manager for our domain by selecting the first option, Choose a certificate from ACM, and then clicking the Request a new certificate from ACM link:
   

   Figure 2.8 – Choose the first option and click the link below to create a new certificate from ACM

4. Once the certificate is created, choose the certificate from the drop-down menu for Certificate name and then click Next.
5. Choose the two security groups, default and jenkins-elb, which we made in Chapter 1, Jenkins Infrastructure with TLS/SSL and Reverse Proxy. Click Next.
6. Create a new target group named jenkins-controller-8080 and change the Port from 80 to 8080 so that it points to the Jenkins controller running on port 8080. Set the Health checks Path to /login. The default / fails the health check because unauthenticated access returns 403 authentication error, whereas /login always succeeds as long as Jenkins is running. Click Next.
7. Add the Jenkins controller instance to registered targets, and then click Next, Create, and Close.
8. Choose the new jenkins load balancer, click the Listeners tab, and then edit HTTP : 80. Delete the current rule of forwarding to jenkins-controller-8080 and replace it with a Redirect to HTTPS at port 443, so that the traffic to HTTP is redirected to HTTPS.

   This is what we should see on the Application ELB Listeners configuration:

   

   Figure 2.9 – Application ELB Listener configuration

9. We can now go to the load balancer's URL (for example, jenkins-1394494507.us-west-2.elb.amazonaws.com) on HTTP or HTTPS to see that Jenkins loads with a TLS certificate error. You can find the load balancer's URL at EC2 Dashboard | Load Balancing | Load Balancers | jenkins | Description | DNS name:
   

   Figure 2.10 – AWS Jenkins on the load balancer URL showing a TLS certificate error

10. Create an A record alias on our domain for Jenkins (for example, jenkins-aws.lvin.ca) and point it to the ELB. Once the DNS cache refreshes, we can go to the Jenkins URL on HTTP or HTTPS and see it load without an error:

Figure 2.11 – AWS Jenkins on the Jenkins URL showing a valid TLS certificate

AWS Jenkins with ELB is up and running using HTTPS.

Setting up an NGINX reverse proxy for the firewalled Jenkins controller

Let's create an NGINX Docker container on the firewalled Jenkins controller host that acts as the reverse proxy for the Jenkins controller Docker container:

1. Copy the TLS certificate and the key that we made using Let's Encrypt (covered in Chapter 1, Jenkins Infrastructure with TLS/SSL and Reverse Proxy), and move them to ~/nginx/certs/ directory.

   robot_acct@firewalled-controller:~$ mkdir -p nginx/certs/

   robot_acct@firewalled-controller:~$ cp ~/letsencrypt/certs/live/jenkins-firewalled
   .lvin.ca/{fullchain.pem,privkey.pem} ~/nginx/certs/

2. Create the ~/nginx/nginx.conf NGINX configuration file as seen in the following snippet. Replace the domains in the two server/server_name sections with your own domains. This file can be downloaded from the book's GitHub repository (https://github.com/PacktPublishing/Jenkins-Administrators-Guide/blob/main/ch2/nginx.conf):

   ~/nginx/nginx.conf

   server {
     listen       80;
     server_name  jenkins-firewalled.lvin.ca;
     return       301  https://$host$request_uri;
   } 

   server {
     listen       443 ssl;
     server_name  jenkins-firewalled.lvin.ca;

     ssl_certificate      /certs/fullchain.pem;
     ssl_certificate_key  /certs/privkey.pem;

     location / {
       proxy_pass          http://localhost:8080;
       proxy_http_version  1.1;

       proxy_set_header  Host  $host:$server_port;
       proxy_set_header  X-Real-IP  $remote_addr;
       proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
       proxy_set_header  X-Forwarded-Proto  $scheme;

       # Max upload size. Useful for a custom plugin.
       client_max_body_size     10m;
       client_body_buffer_size  128k;

       proxy_buffering          off;
       proxy_request_buffering  off;
       proxy_set_header         Connection  "";
     }
   }

3. With the three files in place, start the NGINX container:

   robot_acct@firewalled-controller:~$ docker run \
       --detach \
       --restart on-failure \
       -v ~/nginx/certs:/certs:ro \
       -v ~/nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro \
       --network=host \
       --name nginx \
       nginx

   There are a few flags in this command. Let's examine what they mean:

   • --detach: Run in the background.
   • --restart on-failure: Automatically restart if the container crashes or the machine reboots.
   • -v ~/nginx/certs:/certs:ro: Bind mount the host ~/nginx/certs directory to /certs inside the container so that the TLS certificates are available to the NGINX process. Mount as read-only because NGINX doesn't need to modify them.
   • -v ~/nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro: Bind mount the host ~/nginx/nginx.conf file to /etc/nginx/conf.d/default.conf inside the container so that the configuration file is available to the NGINX process. Mount as read-only because NGINX doesn't need to modify it.
   • --network=host: Allow the container to have direct access to the host's network so that it can manage the traffic for the Jenkins controller container.
   • --name nginx: Name the running container nginx for a memorable name and easier access.
4. We can now go to the NGINX container host's IP (for example, 192.168.1.16) on HTTP or HTTPS to see that Jenkins loads with a TLS certificate error:
   

   Figure 2.12 – Firewalled Jenkins on a reverse proxy IP showing a TLS certificate error

5. Create an A record on our domain for Jenkins (for example, jenkins-firewalled.lvin.ca) and point it to the reverse proxy. Once the DNS cache refreshes, we can go to the Jenkins URL on either HTTP or HTTPS and see it load without an error:
   

Figure 2.13 – Firewalled Jenkins on the Jenkins URL showing a valid TLS certificate

Firewalled Jenkins with an NGINX reverse proxy is up and running using HTTPS.

Terminating the TLS certificate directly on the Jenkins controller

It is also possible to terminate the TLS directly on the Jenkins controller instead of using a reverse proxy. This reduces the amount of infrastructure to manage; however, it is suitable only in a trusted environment for a small-scale Jenkins instance, due to the lack of protection and scaling that a reverse proxy provides. It also lacks the HTTP to HTTPS redirect, as illustrated in the following figure:

Figure 2.14 – Architecture of Jenkins terminating the TLS certificate directly on the controller

Let's begin:

1. Copy the TLS certificate and the key that we made in Chapter 1, Jenkins Infrastructure with TLS/SSL and Reverse Proxy, using Let's Encrypt in the ~/certs/ directory:

   robot_acct@firewalled-controller:~$ mkdir ~/certs/

   robot_acct@firewalled-controller:~$ cp ~/letsencrypt/certs/live/jenkins-firewalled.lvin.ca/{fullchain.pem,privkey.pem} ~/certs/

2. The private key must be converted to an old-styled PKCS #1 format due to an unimplemented feature1. Run the following command to convert the private key:

   robot_acct@firewalled-controller:~$ openssl rsa \
       -in ~/certs/privkey.pem \
       -out ~/certs/privkey.pkcs1.pem

3. Then, restart Jenkins with these updated options:

   robot_acct@firewalled-controller:~$ docker run \
       --detach \
       --restart on-failure \
       -u $(id -u):$(id -g) \
       -v ~/jenkins_home:/var/jenkins_home \
       -v ~/certs:/certs:ro \
    -e JENKINS_OPTS="
    --httpsPort=443
    --httpsCertificate=/certs/fullchain.pem
    --httpsPrivateKey=/certs/privkey.pkcs1.pem" \
    -p 443:443 -p 50000:50000 \
       --name jenkins_controller \
       calvinpark/jenkins:2.263.1-lts

   There are three changes:

   • -v ~/certs:/certs:ro
     Bind mount the ~/certs host directory to /certs inside the container so that the certificates are available to Jenkins. Mount as read-only because Jenkins doesn't need to modify it.
   • -e JENKINS_OPTS="
--httpsPort=443
--httpsCertificate=/certs/fullchain.pem
--httpsPrivateKey=/certs/privkey.pkcs1.pem"
     Configure Jenkins to run on HTTPS using the certificates for TLS termination. Take note that the lines don't end with backslashes – they are a multi-line string.
   • -p 443:443
     Bind HTTPS port 443 on the host to HTTPS port 443 inside the container so that Jenkins is accessible on HTTPS.
4. We can now go to the controller host's IP (for example, 192.168.1.16) on HTTPS to see that Jenkins loads with a TLS certificate error. See Figure 2.10.
5. Create an A record alias on our domain for Jenkins (for example, jenkins-firewalled.lvin.ca) and point it to the controller host's IP. Once the DNS cache refreshes, we can go to the Jenkins URL on HTTPS and see it load without an error. See Figure 2.11.
6. Loading Jenkins on HTTP will not work since we don't have the reverse proxy that redirects HTTP to HTTPS. See Figure 2.15:

Figure 2.15 – Accessing Jenkins using HTTP fails without a reverse proxy

Firewalled Jenkins without a reverse proxy is up and running using HTTPS.