Jenkins Job DSL and Jenkins as IaC
Note: this blog post is adapted from a talk give at a meeting of the Boston Jenkins Area Meetup on Thursday, April 21st, 2016.
Jenkins and IaC
A tool like Jenkins often is the cornerstone of a CI/CD pipeline: it brings together disparate tools like Git, Maven, Packer, and Terraform, providing the “glue” to automate processes and build deployment and testing pipelines. Eventually, the Jenkins server becomes a custom application in its own right, but unfortunately Jenkins does not inherently lend itself to being treated as an application. Jobs are managed manually; testing changes to the pipeline often involves changing them live; and worst of all, losing a Jenkins server can halt developer productivity until backups are restored or, worse still, jobs are recreated from scratch, history being lost in the process.
Here are just a few key differences between IaC best practices and how a Jenkins server is typically managed:
- Infrastructure should be implemented as cattle, not pets. While slaves allow additional capacity to be provisioned, the Jenkins master is the only server holding both business logic and configuration. If the Jenkins master goes down, builds stop.
- Applications should be testable. Having a single-instance application running in a production environment means that changes can never be truly tested in an isolated sandbox. Even a simple task such as upgrading a plugin can have devastating effects on all jobs running on the server.
- Applications should have a clear history. There is an immense deal of value in being able to see why a line of code was added or modified. It’s likewise useful to see what work items such as user stories were involved in getting the application to its current state. Jenkins jobs have no built-in auditing mechanism - it’s not trivial, and often is impossible, to know when, why, or by whom a particular change to a Jenkins job was made. Often it’s not possible to know that a change was made to a job in the first place.
- An application should be easy to maintain and continuously deploy. Woe be the developer who has to manually edit three hundred jobs just to point them at a new Slack channel.
Fortunately, application logic created within Jenkins can be managed just like any other, with code and control.
The Jenkins Job DSL Plugin allows Jenkins jobs to be defined in code - Groovy code, specifically. Code is something that developers are intimately familiar with, and numerous workflows exist to manage. In this case, code will work to demystify what the Jenkins server is doing behind the scenes, provide history and repeatability, and make it simple to make changes to the Jenkins server.
A Git repository, jenkins_jobdsl_demo, contains a Chef cookbook and Vagrant environment suitable for following along with this overview. Simply run the included provision.sh (which assumes working Vagrant and Berkshelf installations) to get started with a Jenkins server on port 8080, and a Gitlab server on port 8000. Jenkins will host jobs; Gitlab will host Job DSL code. Also included are Git, Maven, and the Job DSL and Git plugins for Jenkins.
Setting up Job DSL
Navigate to Gitlab, and create a new repository for Job DSL code to live in called jobdsl. The URL for this repository will be http://localhost:8000/root/jobdsl.git.
After checking out this (empty) repository, create a file called jobdsl.groovy - this file will contain Job DSL commands to provision Jenkins. Its contents at this point should be the absolute base case - a hard-coded shell command to echo back “Hello, World!”.
job('Hello') {
steps {
shell('echo "Hello, World!"')
}
}
This will create a basic (Freestyle) job called “Hello” on the server. Job DSL jobs are broken down into segments based on functional area - in this case, the steps section specifies what steps to run as part of the build. The shell parameter instructs Jenkins to run a shell command, which in this case is an echo Check this file into Git and push it to Gitlab:
[root@app01 jobdsl]$ git add jobdsl.groovy
[root@app01 jobdsl]$ git commit -m "Hello World job."
[master (root-commit) 388c245] Hello World job.
1 file changed, 5 insertions(+)
create mode 100644 jobdsl.groovy
[root@app01 jobdsl]$ git remote add origin http://localhost:8000/root/jobdsl.git
[root@app01 jobdsl]$ git push -u origin master
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 284 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To http://localhost:8000/root/jobdsl.git
* [new branch] master -> master
Branch master set up to track remote branch master from origin.
In Jenkins at http://localhost:8080, create a new Freestyle job named “bootstrap” this job will run Job DSL code from Git. “Process Job DSLs” is now a build step. Point this job at the Gitlab URL previously created, and point the build step at the file in Git which will be checked out into the job’s workspace:
Note also that it’s possible to determine what will happen when Job DSL removes jobs. A removed job is any job which at one time was expressed in Job DSL (Job DSL jobs retain history of what jobs they’ve created), but no longer appears in the code. It’s possible to ignore (e.g. do nothing), disable (to ensure that no further builds run), or delete projects entirely when they are removed from Job DSL. While the default is to ignore removed jobs, selecting Delete enforces tighter management of the server from source control.
After saving and running the bootstrap job, a job called Hello will be created and will be visible on the Jenkins front page:
Running the job will produce a “Hello, World!”:
Started by user anonymous
Building in workspace /var/lib/jenkins/jobs/Hello/workspace
[workspace] $ /bin/sh -xe /tmp/hudson2485962945861355338.sh
+ echo 'Hello, World!'
Hello, World!
Finished: SUCCESS
This is the most basic example of provisioning a Jenkins job through a Git-versioned Job DSL script.
Running builds from Git
Create a new repository in Gitlab called “hello” - the URL for this repository will be http://localhost:8000/root/hello.git. Additionally, create a hello.sh (below) in the root of this repository, mark it as executable, commit it, and push it to Gitlab.
#!/bin/sh
echo "Hello, world!"
Next, instruct Job DSL to check out the Hello job’s workspace from Git, and run the hello.sh script rather than a hard-coded echo:
[root@app01 jobdsl]$ git diff
diff --git a/jobdsl.groovy b/jobdsl.groovy
index e3fcb8d..147ee2a 100644
--- a/jobdsl.groovy
+++ b/jobdsl.groovy
@@ -1,5 +1,8 @@
job('Hello') {
+ scm {
+ git('http://localhost:8000/root/hello.git')<br />+ }
steps {
- shell('echo "Hello, World!"')
+ shell('./hello.sh')
The changes tell Jenkins that the project uses a source code management tool, with the project’s Git URL specified.
Note that a git diff was used to show the changes. The diff to Jenkins has a corresponding diff in Git. Likewise, git log will show a commit detailing what change was made to the server:
[root@app01 jobdsl]$ git log
commit 8e268f80712ed5c46b7c1f70b8dab906501b2aef
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 22:12:24 2016 -0400
Run the hello world script from Git.
commit 388c245d0b19ae52fe17a2960c37c997e9cf4a0e
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 21:33:32 2016 -0400
Hello World job.
Rerun the bootstrap job, and note that the generated job has changed in-place: it now runs the command ./hello.sh rather than echo "Hello, world!", but has kept its build history:
Started by user anonymous
Building in workspace /var/lib/jenkins/jobs/Hello/workspace
Cloning the remote Git repository
Cloning repository http://localhost:8000/root/hello.git
> git init /var/lib/jenkins/jobs/Hello/workspace # timeout=10
Fetching upstream changes from http://localhost:8000/root/hello.git
> git --version # timeout=10
> git fetch --tags --progress http://localhost:8000/root/hello.git +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url http://localhost:8000/root/hello.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url http://localhost:8000/root/hello.git # timeout=10
Fetching upstream changes from http://localhost:8000/root/hello.git
> git fetch --tags --progress http://localhost:8000/root/hello.git +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen 1 remote branch
Checking out Revision 0aa344a68712ff4c6cd78ba166a15014a70e7f3b (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 0aa344a68712ff4c6cd78ba166a15014a70e7f3b
First time build. Skipping changelog.
> git tag -a -f -m Jenkins Build #2 jenkins-Hello-2 # timeout=10
[workspace] $ /bin/sh -xe /tmp/hudson8992155914579787138.sh
+ ./hello.sh
Hello, world!
Finished: SUCCESS
Running builds on checkin
In the last two examples, builds have been manually kicked off. It’s generally preferable to run builds on commit. While Jenkins and Job DSL both have robust support for most SCM tools’ push notifications (GitHub and Gitlab included), the simplest way to do so is to poll Git, in this case once per minute using an SCM cron trigger:
[root@app01 jobdsl]$ git diff
diff --git a/jobdsl.groovy b/jobdsl.groovy
index 147ee2a..9b762c4 100644
--- a/jobdsl.groovy
+++ b/jobdsl.groovy
job('Hello') {
steps {
shell('./hello.sh')
}
+ triggers {
+ scm('* * * * *')
+ }
}
Note that no branch was specified in the Git SCM section. The Git plugin, by default, will build HEAD, meaning that any changes will be picked up for new builds. Create a new branch of the Hello project, use the Jenkins environment variable $GIT_BRANCH to make it clear what branch is running, make the script fail, and check it in:
[root@app01 hello]$ git diff
diff --git a/hello.sh b/hello.sh
index 79a32fd..7de371d 100755
--- a/hello.sh
+++ b/hello.sh
@@ -1,2 +1,3 @@
#!/bin/sh
-echo "Hello, world!"
+echo "Hello from $GIT_BRANCH!"
+exit 1
[root@app01 hello]$ git checkout -b fail
Switched to a new branch 'fail'
[root@app01 hello]$ git commit -a -m "Announce branch name and fail."
[fail 17a034e] Announce branch name and fail.
1 file changed, 2 insertions(+), 1 deletion(-)
[root@app01 hello]$ git push -u origin fail
Counting objects: 3, done.
Writing objects: 100% (3/3), 310 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To http://localhost:8000/root/hello.git
* [new branch] fail -> fail
Branch fail set up to track remote branch fail from origin.
Jenkins will kick off a new build shortly thereafter, which promptly announces that the origin/fail branch is running before failing:
Started by an SCM change
Building in workspace /var/lib/jenkins/jobs/Hello/workspace
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://localhost:8000/root/hello.git # timeout=10
Fetching upstream changes from http://localhost:8000/root/hello.git
> git --version # timeout=10
> git fetch --tags --progress http://localhost:8000/root/hello.git +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/fail
Seen branch in repository origin/master
Seen 2 remote branches
Checking out Revision 17a034e9be7e81a914129f00f1db832c40e8e6fc (origin/fail)
> git config core.sparsecheckout # timeout=10
> git checkout -f 17a034e9be7e81a914129f00f1db832c40e8e6fc
First time build. Skipping changelog.
> git tag -a -f -m Jenkins Build #3 jenkins-Hello-3 # timeout=10
[workspace] $ /bin/sh -xe /tmp/hudson405440234279088245.sh
+ ./hello.sh
Hello from origin/fail!
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Flaming on Failure
In many cases, it’s desirable to send out notifications when builds fail. Job DSL has support for a number of Git plugins such as the e-mail notification and Slack plugins. Setting up an e-mail notification to e-mail both a team and committer on failure is fairly straightforward:
[root@app01 jobdsl]$ git diff
diff --git a/jobdsl.groovy b/jobdsl.groovy
index 9b762c4..20c6066 100644
--- a/jobdsl.groovy
+++ b/jobdsl.groovy
@@ -8,4 +8,7 @@ job('Hello') {
triggers {
scm('* * * * *')
}
+ publishers {
+ mailer('helloteam@localhost', false, true)
+ }
}
The syntax above, per the Job DSL documentation for the mailer resource, instructs Jenkins to always e-mail “helloteam@localhost”, send an e-mail notification for every failed build (not just the first failed build), and to send additional e-mails to individual committers who broke the build.
Merge the failing branch to master to observe Jenkins’ behavior:
[root@app01 hello]$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
[root@app01 hello]$ git merge fail
Updating 0aa344a..17a034e
Fast-forward
hello.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
[root@app01 hello]$ git push
Total 0 (delta 0), reused 0 (delta 0)
To http://localhost:8000/root/hello.git
0aa344a..17a034e master -> master
Started by user anonymous
Building in workspace /var/lib/jenkins/jobs/Hello/workspace
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://localhost:8000/root/hello.git # timeout=10
Fetching upstream changes from http://localhost:8000/root/hello.git
> git --version # timeout=10
> git fetch --tags --progress http://localhost:8000/root/hello.git +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/fail
Seen branch in repository origin/master
Seen 2 remote branches
Checking out Revision 17a034e9be7e81a914129f00f1db832c40e8e6fc (origin/master, origin/fail)
> git config core.sparsecheckout # timeout=10
> git checkout -f 17a034e9be7e81a914129f00f1db832c40e8e6fc
> git rev-list 0aa344a68712ff4c6cd78ba166a15014a70e7f3b # timeout=10
> git rev-list 17a034e9be7e81a914129f00f1db832c40e8e6fc # timeout=10
> git tag -a -f -m Jenkins Build #5 jenkins-Hello-5 # timeout=10
[workspace] $ /bin/sh -xe /tmp/hudson8815884556584754372.sh
+ ./hello.sh
Hello from origin/master!
Build step 'Execute shell' marked build as failure
Sending e-mails to: helloteam@localhost don-code@users.noreply.github.com
Finished: FAILURE
Building the Apache Commons Lang project
Jenkins supports building Maven jobs in addition to freestyle (generic) jobs. Apache Commons Lang is a fairly common open-source, Maven-built project. To build the master branch of Apache Commons using Jenkins’ Maven integration:
[root@app01 jobdsl]$ git diff
diff --git a/jobdsl.groovy b/jobdsl.groovy
index 20c6066..abd8316 100644
--- a/jobdsl.groovy
+++ b/jobdsl.groovy
@@ -12,3 +12,10 @@ job('Hello') {
mailer('helloteam@localhost', false, true)
}
}
+
+mavenJob('commons-lang (master)') {
+ scm {
+ git('https://github.com/apache/commons-lang.git', 'master')
+ }
+ goals('clean package')
+}</span>
Building all branches of Commons Lang
Because Commons Lang is a GitHub project, Job DSL can take advantage of its Groovy roots to interact with the GitHub API to retrieve a list of the project’s branches, then create jobs for each one. Note that the GitHub API could just as easily be output from a service discovery system like Consul, a SQL database, or any other Web API. Additionally, as a JVM language, any third-party dependencies on Jenkins’ classpath can be brought in for use with Job DSL, though doing so is not demonstrated here.
[root@app01 jobdsl]$ git diff
diff --git a/jobdsl.groovy b/jobdsl.groovy
index abd8316..f8f2fdc 100644
--- a/jobdsl.groovy
+++ b/jobdsl.groovy
@@ -1,3 +1,5 @@
+import groovy.json.JsonSlurper
+
job('Hello') {
scm {
git('http://localhost:8000/root/hello.git')
@@ -13,9 +15,12 @@ job('Hello') {
}
}
-mavenJob('commons-lang (master)') {
- scm {
- git('https://github.com/apache/commons-lang.git', 'master')
- }
- goals('clean package')
+def branches = new JsonSlurper().parse(new URL('https://api.github.com/repos/apache/commons-lang/branches').newReader())
+branches.each { branch ->
+ mavenJob("commons-lang (${branch.name})") {
+ scm {
+ git('https://github.com/apache/commons-lang.git', branch.name)
+ }
+ goals('clean install')
+ }
}
Jenkins will create jobs for each of the existing branches that are output from the API. If the Delete option was selected for removed jobs in the bootstrap job creation step, build jobs for branches which no longer exist will be cleaned up from the server each time bootstrap runs. Note that because the master job already existed, a build which was kicked off prior to the inclusion of all other branches will continue to run, unaware of the fact that its definition in Job DSL changed.
A view for Commons Lang
Given that seven jobs were just created wit a single declaration, it’s evident that the Jenkins “All” view can quickly become cluttered. Fortunately, Job DSL supports both Views and Folders within Jenkins for organizing jobs. This final example will demonstrate creating a view keyed from a regular expression which includes only Apache Commons Lang jobs:
[root@app01 jobdsl]$ git diff
diff --git a/jobdsl.groovy b/jobdsl.groovy
index f8f2fdc..3509b50 100644
--- a/jobdsl.groovy
+++ b/jobdsl.groovy
@@ -24,3 +24,18 @@ branches.each { branch ->
goals('clean install')
}
}
+
+listView('Apache Commons Lang') {
+ jobs {
+ regex(/commons-lang.*/)
+ }
+ columns {
+ status()
+ weather()
+ name()
+ lastSuccess()
+ lastFailure()
+ lastDuration()
+ buildButton()
+ }
+}
Conclusion
Job DSL effectively solves the problems enumerated at the beginning of this post. First and foremost, as the state of the Jenkins server is now fully managed through code, a loss of the server is no longer catastrophic - in fact, an organization may become comfortable with fully ephemeral Jenkins servers: ones which have no expectation of persistent state, and can come and go at will.
Because Jenkins’ jobs are defined as code, it’s now fully possible to work on them in feature branches, test them by standing that branch up against a blank server, and so forth. This significantly reduces risk when making changes to Jenkins.
The entire known state of the server is now defined in code. Furthermore, there is a verbose history showing exactly what has changed and when, which is visible using common tools like git blame and git log; for instance:
[root@app01 jobdsl]$ git log
commit a6a83bd30e135aa4279a38381d181f79cb989145
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 23:44:12 2016 -0400
Create a view for Commons Lang projects.
commit 0f235d26bd4c47b1fb9288ab7400d9f70afada4c
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 23:37:16 2016 -0400
Build all branches of Commons Lang.
commit 0fe9dc6721e7fe3424290caba1279b09fdf5562a
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 23:24:27 2016 -0400
Build the master branch of Commons Lang.
commit 72f736a793ac52a4a2551deaa327593c47c24541
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 23:08:45 2016 -0400
Flame committers who failed the build.
commit eeecc29b3c3f0d33536de309936c0594412caa27
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 22:30:49 2016 -0400
Poll Git for changes to the hello job.
commit 8e268f80712ed5c46b7c1f70b8dab906501b2aef
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 22:12:24 2016 -0400
Run the hello world script from Git.
commit 388c245d0b19ae52fe17a2960c37c997e9cf4a0e
Author: don-code <don-code@users.noreply.github.com>
Date: Sat Apr 23 21:33:32 2016 -0400
Hello World job.
Finally, using Job DSL’s underlying Groovy syntax, it’s possible to bulk create and bulk update jobs en masse, from a single configuration file, without spending time manually maintaining the Jenkins server.
A final word on automation: Job DSL only manages jobs. The Vagrant box linked at the beginning of this post makes use of the Jenkins Chef cookbook, which allows for from-scratch master and slave provisioning, installation of plugins, and configuration of the bootstrap job, which otherwise would have been the single manually configured job of the server.