Attaching SSH and inbound agents

We have two Jenkins agents, one on AWS and another inside the corporate firewall. Each agent will connect to both Jenkins controllers, one on AWS and another inside the firewall.

The AWS agent is accessible by both the AWS controller and the firewalled controller. We will connect the AWS agent as an SSH agent, which is simple and effective:

Figure 2.20 – Architecture of SSH and inbound agents

The firewalled agent, on the other hand, is not accessible by the AWS controller. The only way for them to communicate is by the agent initiating the connection, so we will connect the agent as an inbound agent. The firewalled controller can access the firewalled agent, therefore it will be connected as an SSH agent.

SSH agent

An SSH agent is the most widely used agent type for an agent running a Unix-based operating system. In our environment, there are three SSH agent connections, and the setup process is the same for all three:

• AWS controller → AWS agent
• Firewalled controller → AWS agent
• Firewalled controller → Firewalled agent

Let's begin:

1. An SSH agent host is a VM running Ubuntu 20.04, just like the controllers. We have already installed Docker on it in Chapter 1, Jenkins Infrastructure with TLS/SSL and Reverse Proxy. SSH into the agent and install JDK 11:

   agent:~$ sudo apt update

   agent:~$ sudo apt install -y openjdk-11-jdk

2. Still on the agent, use ssh-keygen to create an SSH public/private key pair, and have the current user (ubuntu on AWS agent and robot_acct on the firewalled agent) accept the key. Don't forget to set the correct file permission on the ~/.ssh/authorized_keys file:

   agent:~$ ssh-keygen 

   agent:~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

   agent:~$ chmod 600 ~/.ssh/authorized_keys 

   agent:~$ ls -la ~/.ssh

   total 20
   drwx------ 2 ubuntu 4096 Dec 15 04:34 .
   drwxr-xr-x 4 ubuntu 4096 Dec 15 04:31 ..
   -rw------- 1 ubuntu  577 Dec 15 04:36 authorized_keys
   -rw------- 1 ubuntu 2610 Dec 15 04:34 id_rsa
   -rw-r--r-- 1 ubuntu  577 Dec 15 04:34 id_rsa.pub

3. We need to store the private key in the Jenkins credential store so that the controller can use it to connect to the agent. Output the content of the private key and keep it handy:

   agent:~$ cat ~/.ssh/id_rsa

   -----BEGIN OPENSSH PRIVATE KEY-----
   b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABl
   [...]
   WM1+k8b+6GZJMAAAAXdWJ1bnR1QGlwLTE3Mi0zMS0xNS0xOTEBAgME
   -----END OPENSSH PRIVATE KEY-----

4. Go to Global Credentials and then click Add Credentials:
   • Kind: SSH Username with private key
   • Scope: System (Jenkins and nodes only)
   • ID: aws-agent-ubuntu-priv or firewalled-agent-robot_acct-priv
   • Description: Same as ID
   • Username: ubuntu or robot_acct
   • Private Key: Check the radio button for Enter directly and click Add. The gray box turns into a textbox – copy and paste the private key into this box.
   • Passphrase: Enter if you've created the SSH keys with a passphrase:

Figure 2.21 – Adding an SSH private key to the Jenkins credential store

5. Click OK to save. The SSH user and the private key are now stored as a secret in the Jenkins credential store:
   

   Figure 2.22 – SSH private key for the Ubuntu user in the AWS agent saved as a credential

6. We also need to get the agent's SSH host key because SSHing to a host for the first time requires that you accept the host key. Print the host key from the agent and keep it handy as we will use it soon:

   agent:~$ cat /etc/ssh/ssh_host_rsa_key.pub

   ssh-rsa AAAAB3N[...]iQwaAqsEDN1e+c= root@ip-172-31-15-191

7. Now, let's add the agent to the controller using the SSH keys. Go back to the Jenkins home page by clicking the Jenkins icon in the upper-left corner, and then click Manage Jenkins | Manage Nodes and Clouds | New Node.
8. Enter Node name, choose the Permanent Agent radio button, and click OK. Populate the following fields:
   • Name: aws-aws-agent, firewalled-aws-agent, or firewalled-firewalled-agent
   • # of executors: 10 (this allows one agent to handle 10 concurrent builds. Since each build is isolated in a Docker container, there isn't a concern of concurrently running builds stepping on each other)
   • Remote root directory:
     • AWS controller → AWS agent: /home/ubuntu/aws-aws-agent
     • Firewalled controller → AWS agent: /home/ubuntu/firewalled-aws-agent
     • Firewalled controller → Firewalled agent: /home/robot_acct/firewalled-firewalled-agent
   • Labels: docker
   • Launch method: Launch agents via SSH
   • Host: IP of the agent
   • Credentials: <user> (<env>-agent-<user>-priv)
   • Host Key Verification Strategy: Manually provided key Verification Strategy
   • SSH Key: Copy and paste the host key into the box

   This is what it should look like:

   

   Figure 2.23 – SSH agent creation page

9. Save to create the agent. In a few seconds, the SSH agent will be connected. Click the agent name and then Log to see an output similar to this:

SSHLauncher{host='172.31.38.41', port=22, credentialsId='aws-agent-ubuntu-priv', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyProvidedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}

[01/18/21 04:12:03] [SSH] Opening SSH connection to 172.31.38.41:22.

[01/18/21 04:12:03] [SSH] SSH host key matched the key required for this connection. Connection will be allowed.

[01/18/21 04:12:03] [SSH] Authentication successful.

[...]

[01/18/21 04:12:04] [SSH] Starting agent process: cd "/home/ubuntu/aws-aws-agent" && java -jar remoting.jar -workDir /home/ubuntu/aws-aws-agent -jar-cache /home/ubuntu/aws-aws-agent/remoting/jarCache

[...]

<===[JENKINS REMOTING CAPACITY]===>channel started

Remoting version: 4.5

This is a Unix agent

Evacuated stdout

Agent successfully connected and online

Congratulations! We have an agent and we can run builds now!

Inbound agent

An inbound agent allows us to add an agent even when the controller can't reach the agent due to a network restriction such as a firewall, but the agent can reach the controller. As the name suggests, an inbound agent initiates the connection from the agent to the controller, which is the opposite of the SSH agent, where the controller initiates the connection.

An inbound agent host is a VM running Ubuntu 20.04, just like the controllers and the SSH agents. Let's configure it:

1. We have already installed Docker on it in Chapter 1, Jenkins Infrastructure with TLS/SSL and Reverse Proxy. SSH into the agent and install JDK 11:

   robot_acct@firewalled-agent:~$ sudo apt update

   robot_acct@firewalled-agent:~$ sudo apt install -y openjdk-11-jdk

2. Next, create the work directory and download agent.jar from our Jenkins. The address for agent.jar is https://<Jenkins URL>/jnlpJars/agent.jar:

   robot_acct@firewalled-agent:~$ mkdir inbound-agent

   robot_acct@firewalled-agent:~$ cd inbound-agent/

   robot_acct@firewalled-agent:~/inbound-agent$ wget https://jenkins-aws.lvin.ca/jnlpJars/agent.jar

3. Go to Jenkins to create a placeholder for the inbound agent. Click Manage Jenkins | Manage Node and Clouds | New Node. Enter inbound-agent in the Node name field, choose the Permanent Agent radio button, click OK, and then populate the following fields:
   • Name: inbound-agent
   • Remote root directory: /home/robot_acct/inbound-agent
   • # of executors: 10 (this allows one agent to handle 10 concurrent builds. Since each build is isolated into a Docker container, there isn't the worry of concurrently running builds stepping on each other.)
   • Labels: docker
   • Launch method: Launch agent by connecting it to the master
   • Advanced | Tunnel connection through: <public IP of the controller>:50000 (In my case, it's 52.88.1.104:50000. Recall that in the AWS: FAQs, routing rules, EC2 instances, and EIPs section of Chapter 1, we opened the port 50000 on the EC2 instances directly, instead of on the ELB. Figure 1.6 and Figure 1.7 illustrate these configurations. In order for the inbound agent to connect to Jenkins on port 50000, we need to route the traffic through the controller.)

Figure 2.24 – Inbound agent creation page

4. Click Save and then click inbound-agent with the red x on the icon. The page shows us three different ways to connect:

Figure 2.25 – Inbound agent connection options

We will use the third method. Simply copy and paste the two commands into the CLI on the inbound agent. Here is the abridged output:

robot_acct@firewalled-agent:~/inbound-agent$ echo 2677db7766582989f1f2333bceb056d474a56e88c793f03a795b4192cf782db6 > secret-file

robot_acct@firewalled-agent:~/inbound-agent$ java -jar agent.jar -jnlpUrl https://jenkins-aws.lvin.ca/computer/inbound-agent/slave-agent.jnlp -secret @secret-file -workDir "/home/robot_acct/inbound-agent"

INFO: Using /home/robot_acct/inbound-agent/remoting as a remoting work directory

[...]

INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check

INFO: Agent discovery successful
  Agent address: 52.88.1.104
  Agent port:    50000
  Identity:      61:2a:ef:73:90:8d:40:ed:01:0d:c9:13:ee:76:f4

INFO: Handshaking

INFO: Connecting to 52.88.1.104:50000

INFO: Trying protocol: JNLP4-connect

INFO: Remote identity confirmed: 61:2a:ef:73:90:8d:40:ed:01:0d:c9:13:ee:76:f4:26

INFO: Connected

And that's all! Now you have a Jenkins instance that crosses the corporate firewall. You can refresh the Jenkins agent page to see that the agent is connected.

The agent connection is running in the foreground of the CLI. Stop the connection with Ctrl + C, and then run the command with the trailing ampersand (&) to send it to the background:

^Crobot_acct@firewalled-agent:~/inbound-agent$ java -jar agent.jar -jnlpUrl https://jenkins-aws.lvin.ca/computer/inbound-agent/slave-agent.jnlp -secret @secret-file -workDir "/home/robot_acct/inbound-agent" &

[1] 8110

robot_acct@firewalled-agent:~/inbound-agent$ Dec 28, 2020 7:48:53 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir

INFO: Using /home/robot_acct/inbound-agent/remoting as a remoting work directory

[...]

INFO: Connected

robot_acct@firewalled-agent:~/inbound-agent$ 

You can now close the SSH connection to the agent and the agent will stay connected to the controller.

Let's now see how we can use the systemd service to make this process restart when the VM reboots.

Creating a systemd service to auto-connect the agent

When the inbound agent connection process dies, perhaps due to a network or a memory issue, it needs to be reconnected. We can have the agent reconnect automatically by creating a service.

Create a new file in /etc/systemd/system/jenkins-inbound-agent.service with the following content. You can download this file from the book's GitHub repository

/etc/systemd/system/jenkins-inbound-agent.service

[Unit]
Description=Jenkins Inbound Agent
Wants=network.target
After=network.target

[Service]
ExecStart=java -jar /home/robot_acct/inbound-agent/agent.jar -jnlpUrl https://jenkins-aws.lvin.ca/computer/inbound-agent/slave-agent.jnlp -secret @/home/robot_acct/inbound-agent/secret-file -workDir /home/robot_acct/inbound-agent

User=robot_acct
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Once the file is created, start the service to kick off the process, and then enable the service so that the service restarts upon a VM reboot. You can see the same console output in the logs using the journalctl command:

robot_acct@firewalled-agent:~$ sudo systemctl start jenkins-inbound-agent

robot_acct@firewalled-agent:~$ sudo systemctl enable jenkins-inbound-agent

robot_acct@firewalled-agent:~$ sudo journalctl -u jenkins-inbound-agent

The inbound agent is now configured to auto-reconnect upon a failure. Try rebooting the VM to see that the inbound agent reconnects automatically.

There are additional ways to connect an inbound agent. See the online docs (https://github.com/jenkinsci/remoting/blob/master/docs/inbound-agent.md) for the latest information.

Labels and Usage

Here is some more information about the Labels and Usage fields on the agent configuration page. Let's talk about Labels first.

Agent labels

In the Labels textbox, we can put multiple keywords delimited by a space to describe an agent. Suppose these four agents have the following labels:

• centos8-agent: docker linux centos centos8
• ubuntu2004-agent: docker linux ubuntu ubuntu2004
• windows-95-agent: windows windows95
• windows-10-agent: docker windows windows10

A Jenkinsfile can specify one or more labels in the agent label directive to specify the agent that it requires. Builds for a pipeline that specifies agent { label 'docker' } would run on centos8-agent, ubuntu2004-agent, or windows-10-agent because the three agents have the docker label. Builds for a pipeline that specifies agent { label 'windows' } would run on any of the two Windows agents. In a more advanced use case, builds for a pipeline that specifies agent { label 'windows && docker' } would run only on windows-10-agent because that is the only agent with both windows and docker labels. Similarly, builds for a pipeline that specifies agent { label 'centos || ubuntu' } would run on any of the two Linux agents.

The agent labels were more useful before Docker was invented – in those dark days, each agent would be a VM with a specific set of tools preinstalled, and labels were used to identify its operating system, tools, and configurations. With Docker available, nearly all agents can have just one docker label as we have configured, and each pipeline can specify its requirements in the Dockerfile. Labels are, of course, still useful if you are using bare-metal agents (rather than Docker or Dockerfile agent) with specific hardware characteristics such as GPU availability or a non-x86 CPU architecture – a pipeline can request an agent with specific hardware using a label.

One last thing about agent labels is that each agent's own name acts as a label. For example, builds for a pipeline that specifies agent { label 'ubuntu2004-agent' } would run only on ubuntu2004-agent, even if you didn't label the agent with its own name. If you need to pin a pipeline to a specific agent, simply use the agent's name as the label.

Next, let's look at Usage.

Agent usage

Usage configuration determines agent availability. In the previous section, we saw that a Jenkinsfile can request an agent with a specific set of labels. A Jenkinsfile can also request any available agent without any label specifications with agent any. In a Jenkins environment with a homogenous set of agents (such as all of them being Ubuntu 20.04), a simple operation such as a file copy doesn't really need a special capability that's noted by agent labels. Usage configuration of an agent determines whether the agent is eligible to be used by a pipeline that specifies agent any.

Choosing Use this node as much as possible makes the agent available to the builds for a pipeline with agent any.

Choosing Only build jobs with label expressions matching this node makes the agent unavailable to the builds for a pipeline with agent any. As the option message says, only build jobs with label expressions matching this node (as opposed to a build job requesting any agent) will run on this node. It's important to understand that this doesn't stop a build from running on this agent if a pipeline pins to this agent. Use the Job Restrictions plugin if you need to limit which pipelines can use an agent.

Also, take note that at least one agent must be configured with the Use this node as much as possible option for a pipeline with agent any to work. If you are setting up a Docker Cloud, a Docker Agent Template is usually a good choice for this configuration.

Let's continue to create a Docker Cloud.