Jenkins As Code

So far in our engineering blog, we have seen a couple of different technologies and features, such as Hashicorp’s Vault and Kubernetes Autoscaling. This time, we are going back to the basics to set up our CI/CD pipeline with Jenkins.

I think it is safe to assume that most DevOps engineers out there are already familiar with Jenkins, but, just in case, this is how Jenkins describes itself on its homepage:

“The leading open source automation server, Jenkins provides hundreds of plugins to support building, deploying, and automating any project.”

So, in a few words, it is a CI/CD tool. Now back to the point.

We were recently tasked with migrating our Jenkins instance to a different AWS account in accordance with AWS’ best practices. Being hardcore fans of Infrastructure as Code (IaC), we wanted to do this the right way, the Terragrunt way.

So what is the setup we are looking for?

1. Jenkins can be a mess of plugins and dependencies. Having its initial setup documented in code would make it easy to see what is used and why.
2. No UI setups. It is not that we hate UIs but changing behaviours and components with a few clicks often goes undocumented. This means that down the road, engineers will be scratching their head trying to figure out the setup. A phrase that often comes to mind is:
   “When I set this up, only God and I understood what it did… Now only God knows.”
3. Nothing beats the feeling of provisioning infrastructure with a couple of command line commands.

We saw three main benefits to this approach:

• Since we are working on AWS and we don’t want Jenkins to live inside our Kubernetes cluster, we would need an AMI of the Jenkins server deployed via an Auto Scaling Group. This would allow us to roll out new versions of the Jenkins server with almost zero downtime.
• We would need to find a way to provision Jenkins configuration (repositories, jobs, credentials, and plugin settings) without visiting the Jenkins UI at all. This configuration should be loaded when launching the Jenkins server as well as when we update the configuration.

Building the AMI

Since we knew that Jenkins is a common tool in the industry, we searched for public Jenkins AMIs. It didn’t take us long to agree on using Bitnami’s Jenkins AMI.

Of course, just deploying the specific image was not good enough. We wanted all the plugins we would use already installed, among other things.

To repackage the AMI, we used Packer, another one of Hashicorp’s tools. We also wanted to integrate this process into our Terragrunt code. In the end, we came up with the following file structure in our Terragrunt repository:

Running terragrunt apply in the jenkins-master folder should create the AMI and expose its attributes so it can be used as a dependency in other modules, like in the Auto Scaling Group.

To achieve this in the terragrunt.hcl file we used a simple module that implements Terraform’s AMI data source. The important step, though, was to implement a “before_hook” that builds the AMI before the module would retrieve and expose it.

The contents of the file ended up looking like this:

terragrunt.hcl

terraform {
 source = "git::git@github.com:transifex/terraform-modules.git//modules/data-ami?ref=v0.1.44"
 before_hook "before_hook" {
   commands = ["apply"]
   execute = [
     "packer",
     "build",
     "-var", "assumed_role=${local.iam_role}",
     "-var", "vpc_id=${local.vpc_id}",
     "-var", "subnet_id=${local.public_subnet}",
     "image/main.pkr.hcl",
   ]
 }
}
 
locals {
 # These can / should be loaded as dependencies from other modules
 iam_role = "arn:aws:iam::xxxxxxxx:role/my_role"
 public_subnet = "vpc-xxxxxxx"
 public_subnet = "sub-xxxxxxx"
}
 
include {
 path = find_in_parent_folders()
}
 
inputs = {
 filter_names = ["jenkins-master"]
}

The image folder would hold the Packer configuration and the provisioning script we wrote.

setup.sh

#!/bin/bash
## Install apt packages that are needed or wanted
apt-get install locate && updatedb
 
## Wait for jenkins to start
function wait_for_jenkins()
{
 until $(curl --output /dev/null --silent --head --fail http://localhost/jnlpJars/jenkins-cli.jar); do
   printf '.'
   sleep 5
 done
 echo "Jenkins launched"
}
wait_for_jenkins
 
## Change shell of jenkins user
usermod --shell /bin/bash jenkins
 
## Download the jenkins-cli.jar
su - jenkins -c "wget localhost/jnlpJars/jenkins-cli.jar -O /opt/bitnami/jenkins/jenkins_home/jenkins-cli.jar"
 
## Retrieve the default admin user password
userpass=$(cat /home/bitnami/bitnami_credentials | grep 'The default username and password is' | rev | cut -d' ' -f1 | cut -d'.' -f2 | sed "s/'//g" | rev)
 
## Install Jenkins plugins that are needed (and restart Jenkins)
/opt/bitnami/java/bin/java -jar /opt/bitnami/jenkins/jenkins_home/jenkins-cli.jar -s http://localhost:8080/ -auth user:$userpass install-plugin configuration-as-code blueocean ec2 slack github-autostatus metrics github-pullrequest role-strategy google-login job-dsl basic-branch-build-strategies parameterized-scheduler

main.pkr.hcl

variable "vpc_id" {
 type = string
}
 
variable "subnet_id" {
 type = string
}
 
variable "assumed_role" {
 type = string
}
 
source "amazon-ebs" "jenkins-master" {
 assume_role {
   role_arn = "${var.assumed_role}"
 }
 
 ami_name                = "jenkins-master"
 instance_type           = "t2.medium"
 region                  = "eu-west-1"
 source_ami              = "ami-0562a5f132869dd5a"
 ssh_username            = "bitnami"
 ami_description         = "Jenkins Master AMI"
 shutdown_behavior       = "terminate"
 spot_price              = "auto"
 spot_price_auto_product = "Linux/UNIX"
 
 ssh_pty               = true
 force_delete_snapshot = true
 force_deregister      = true
 
 // Vpc && subnet to launch the spot instance
 vpc_id    = "${var.vpc_id}"
 subnet_id = "${var.subnet_id}"
 
 // Associate public ip so packer can ssh to provision
 associate_public_ip_address = true
 // Use the public IP to connect
 ssh_interface = "public_ip"
}
 
 
build {
 sources = ["source.amazon-ebs.jenkins-master"]
 provisioner "shell" {
   execute_command = "echo '' | sudo -S su - root -c '{{ .Vars }} {{ .Path }}'"
   script          = "image/setup.sh"
 }
}

Et voilà!

With a simple `terragrunt apply` command, we can build our image, and we are able to load its attributes in future modules.

Jenkins Configuration As Code

In our search of ways to automate Jenkins provisioning (and after a couple of hours struggling with groovy scripts) we came across the following plugin: JCaC ( Jenkins Configuration as Code). And yes it is what the name implies. A plugin adopted by the Jenkins community whose sole purpose is to provide the ability to define Jenkins and plugin configuration in a YAML file.

This file can be placed inside the Jenkins server or loaded from url. As a cherry on top, you could have your Jenkins instance reload the configuration file by hitting a specific endpoint.

We quickly took to the task and started configuring our Jenkins and plugins using the YAML format. The end file is too large and too specific to our needs to share it here but you can find numerous examples on how to set it up in the demos provided by JCaC.

Alternatively, if you are currently running Jenkins you can install the plugin and export your current configuration.  Once we finished creating the configuration file in order to make it available to Jenkins we decided to add it in a private S3 bucket that only Jenkins can access. As always we used Terragrunt for this.Two items remained now:

1. Have Jenkins reload the file when a change was made
2. Provision the file url in the initialization of Jenkins so it can be discovered on startup.

For the first item it was as simple as adding an “after_hook” in the “terragrunt apply” that was uploading the file to s3:

terragrunt.hcl

terraform {
 source = "git::git@github.com:terraform-aws-modules/terraform-aws-s3-bucket.git//modules/object?ref=v1.25.0"
 
 after_hook "after_hook" {
   commands     = ["apply"]
   execute      = ["curl", "-X", "POST", "jenkins.domain/reload-configuration-as-code/?casc-reload-token=xxxxxxxx"
 }
}

...

This means that every time we update the config file, a post would be made to Jenkins reloading the configuration.

In order to make sure that Jenkins will load the file on initialization as well, we went back to the AMI we built in the previous step and we changed the provisioning script to provide the necessary configuration:

main.pkr.hcl

...
build {
 sources = ["source.amazon-ebs.jenkins-master"]
 provisioner "shell" {
   environment_vars = [
     "jenkins_casc_token=${var.jenkins_casc_token}",
     "vpc_endpoint=${var.vpc_endpoint}"
   ]
   execute_command = "echo '' | sudo -S su - root -c '{{ .Vars }} {{ .Path }}'"
   script          = "image/setup.sh"
 }
}

setup.sh

...
## Jenkins configuration as code options
sed "s,\"javaOpts\": \"\",\"javaOpts\": \"-Dcasc.reload.token=$jenkins_casc_token -Dcasc.jenkins.config=$vpc_endpoint\",g" -i /root/.nami/registry.json

terragrunt.hcl

terraform {
 source = "git::git@github.com:transifex/terraform-modules.git//modules/data-ami?ref=v0.1.44"
 before_hook "before_hook" {
   commands = ["apply"]
   execute = [
     "packer",
     "build",
     "-var", "assumed_role=${local.iam_role}",
     "-var", "vpc_id=${local.vpc_id}",
     "-var", "subnet_id=${local.public_subnet}",
     "-var", "jenkins_casc_token=${local.casc_token"]}",
     "-var", "vpc_endpoint=${local.s3_endpoint}",
     "image/main.pkr.hcl",
   ]
 }
}
 
locals {
 # These can / should be loaded as dependencies from other modules
 iam_role = "arn:aws:iam::xxxxxxxx:role/my_role"
 public_subnet = "vpc-xxxxxxx"
 public_subnet = "sub-xxxxxxx"
 casc_token = "xxxxxxxxxxx"
 s3_endpoint = "xxxxxxxxxxx"
}
 
...

And now, with those changes, the configuration file can be loaded on startup!

Seeding Your Jobs

While the CasC solved most of our configuration problems, one thing it could not do, at least not by itself, was creating our Jenkins jobs. To achieve this, you would need another plugin, job-dsl, that can work in tandem with CasC.

job-dsl allows you to define jobs in a programmatic manned inside the Jenkins configuration file created earlier. I would be remiss if I did not mention here that this is not for the faint of heart. While with this plugin, you can create amazing automation, the documentation is lacking in many places and requires a lot of trial and error to achieve good results.

In our case, we wanted to provide our github organization and some hand-picked repositories that contain their own Jenkinsfiles. To achieve this, we used an OrganizationFolder. This is the script we used (located in the YAML file along with the rest of our configuration).

jobs:
 - script: >
     organizationFolder('Transifex') {
       description("Transifex organization folder configured with JCasC")
       displayName('Transifex')
       buildStrategies {
         buildChangeRequests {
           ignoreTargetOnlyChanges(true)
           ignoreUntrustedChanges(true)
         }
         skipInitialBuildOnFirstBranchIndexing()
       }
       triggers {}
       // "Projects"
       organizations {
         github {
           repoOwner("transifex")
           credentialsId("xxxxxxx")
           traits {
             // Discovers branches on the repository.
             gitHubBranchDiscovery {
               // Determines which branches are discovered.
               // 1 = Exclude branches that are also filed as PRs
               strategyId(1)
             }
             gitHubPullRequestDiscovery {
               // Determines how pull requests are discovered: Merging the pull request with the current target branch revision Discover each pull request once with the discovered revision corresponding to the result of merging with the current revision of the target branch.
               // 1 = Merging the pull request with the current target branch revision
               strategyId(1)
             }
             gitHubExcludeArchivedRepositories()
             sourceWildcardFilter {
               // Space-separated list of project (repository) name patterns to consider.
               includes("xxxxx yyyyyy zzzzzz")
               excludes("")
             }
           }
         }
       }
     }
 
...

Final Notes

All in all, we have to admit that provisioning Jenkins with IaC principles was a tiresome experience for our team. A lot of items were not or were poorly documented, and the answer to our problems was never obvious. But once it was done, it was easy to alter and keep track of it.

Say we needed a new Jenkins in the future, spinning it up would be as simple as writing a few lines and editing a couple of files. And all of this would be forever documented in our repository. We hope this article will provide a guiding light for the brave engineers that dare to venture into these paths in the future.