Docker : Jenkins Multibranch Pipeline with Jenkinsfile and Github

bogotobogo.com site search:

Official Jenkins image

In this post, we'll setup Jenkins from scratch in a Docker container. Then, we'll create a Multibranch pipeline with git that runs a Jenkinsfile.

The official Jenkins image for Docker is available from https://hub.docker.com/r/jenkins/jenkins/.

This is a fully functional Jenkins server, based on the weekly and LTS releases.

To use the latest LTS: docker pull jenkins/jenkins:lts.

Usage

To run container in a detached mode and to create a 'jenkins_home' docker volume attached to the host machine volume as shown in https://github.com/jenkinsci/docker/blob/master/README.md, we do the fillowing:

$ docker run -d -v jenkins_home:/var/jenkins_home -p 8080:8080 -p 5000:5000 jenkins/jenkins:lts
...
hudson.WebAppMain$3#run: Jenkins is fully up and running

$ docker ps
CONTAINER ID  IMAGE                COMMAND                 CREATED        STATUS        PORTS                                              NAMES
666101198dd0  jenkins/jenkins:lts  "/sbin/tini -- /usr/…"  2 minutes ago  Up 2 minutes  0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   dazzling_euclid

The mapping (i.e. "publish") port 8080 of the current container to port 8080 on the host machine. The first number represents the port on the host while the last represents the container's port.
This is optional but we map port 5000 of the current container to port 5000 on the host machine. This is only necessary if we have set up one or more inbound Jenkins agents on other machines, which in turn interact with our jenkins container (the Jenkins "controller"). Inbound Jenkins agents communicate with the Jenkins controller through TCP port 50000 by default.
We're also mapping the /var/jenkins_home directory in the container to the Docker volume with the name jenkins_home. Instead of mapping the /var/jenkins_home directory to a Docker volume, we could also map this directory to one on our machine's local file system. For example, specifying the option --volume $HOME/jenkins:/var/jenkins_home would map the container's /var/jenkins_home directory to the jenkins subdirectory within the $HOME directory on our local machine, which would typically be /Users/<username>/jenkins or /home/<username>/jenkins.

Thanks to the volume, we will survive the container stop/restart/deletion. Also, by making an explicit volume, we can manage it and attach to another container for upgrades.

Note: On a Mac host, the volume, jenkins_home, is located on vm where the docker in running:

$ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

docker-desktop:~# ls /var/lib/docker/volumes
...
jenkins_home
...

We can access our Jenkins running on the container:

Let's go into the running Jenkins container and retrieve an initial password to unlock Jenkins:

$ docker exec -i -t --user root 666101198dd0 /bin/bash
root@666101198dd0:/# 

root@666101198dd0:/# cat /var/jenkins_home/secrets/initialAdminPassword
56316ef50c714872a210ee44564cbe40

As instructed, paste the password and click "Continue":

After unlocking Jenkins, the Customize Jenkins page appears. Here we can install any number of useful plugins as part of our initial setup.

Select "Install suggested plugins", then the setup wizard shows the progression of Jenkins being configured:

After customizing Jenkins with plugins, Jenkins asks us to create our first administrator user:

Click "Save and Finish" and "Start using Jenkins":

Credentials in Jenkins

Credentials in Jenkins have three scopes depending where this credential can be used:

Global(for Jenkins instances such as email auth, slave connection, etc) scope
System (job related) scope

Per project

For our Multibranch project, the following needs to be set:

Note that we used PAT (Personal Access Token) for the password.

Here is the git repo.

Here are the credentials so far.

As we can see the credentials for the 'My Pipeline" won't be visible to other project. Each project can have its own credentials that are not visible to other projects.

Note also that the System credentials are hidden from our project, 'My Pipeline", and we only see the Jenkins/Global scope.

Multibranch pipeline

Configure the project with the credentials we put in the previous section:

Hit "Save", then it will run. From the "Console Output", we see it was successfully built:

As the project name suggests the Multibranch Pipeline can do several branches not just the master branch as in the previous example. In the project configuration page, we can set the branches we want by adding filter for the behavior of repo discovery:

The added filter should look like this:

Note that we have another option to use the filter by name with regular expression (for example, we can use ".*" to apply to all branches).

Hit "Save" button, then w'll have the following "Console Output" that ran against the release branch with a newly updated Jenkinsfile:

We can also check it from "Scan Repository Log":

Jenkinsfile - Pipeline syntax

Jenkinsfile is a script that defines a pipeline configuration instead of using Jenkins UI. So, simply put, it is a "Pipeline as a Code".

The Jenkinsfile Pipeline syntax can be found in Pipeline Syntax.

There are two types of Jenkinsfile:

Declarative Pipeline - we''ll use're using it in this post.
Scripted Pipeline - it is effectively a general-purpose DSL built with Groovy.

Here is a basic Jenkins file:

pipeline {
    agent any
    stages {
        stage('build') {
            steps {
                echo 'Hello world, this is multibranch pipeline for Dev branch'
            }
        }
        stage('test') {
            steps {
                echo 'testing Dev...'
            }
        }
        stage('deploy') {
            steps {
                echo 'deploying Dev...'
            }
        }
    }
}

Here are the requiremed fields of a Jenkinsfile:

"pipeline" - should be top level
"agent" - where to execute
"stages" - where the works are done
"stage" and "steps"

We also need to add the "dev" branch to our pipeline conf:

Then, click "Save" to run the Jenkinsfile.

Replaying Jenkinsfile

What if we want to test the Jenkinsfile without actually commit it to our repo?

Here is a "Replay" feature in Jenkins.

Just go back to the dev build and select "Replay":

It will show our Jenkinsfile and we just want to add Groovy script withing the "build" step:

Hit "Run":

As we can see, the logs show that the run was successful!

Restart from Stage

There is another feature that we can specify stages. For example, in our case we have three stages: build, test, deploy.

Let's just do the 'deploy" skipping the prevous other two stages:

Now we can build the "deploy" stage only as shown below:

Git Integration

To trigger Jenkins job automatically, we may want to setup for push notification and polling for changes in the Git.

Push notification - Version Control notifies Jenkins on a new commit. More efficient than polling by Jenkins.
Polling - Jenkins itself polls Git in regular intervals.

(Optional) Github-webhook to local machine via ngrok

Since we're working on local environmanet, the Github cannot connect to our Jenkins. So, we need a proxy that give us a public ip by doing remote port forawrding.

Follow the steps in https://dashboard.ngrok.com/get-started/setup.

$ ./ngrok http 8080

then, we get the following:

We're going to use https://f4fd3444a3b2.ngrok.io as our public ip from now on.

Now, while the ngrok terminal opened, let's setup a webhook.

Navigate to our Github repository (https://github.com/Einsteinish/webhook-demo.git) that we want Jenkins to monitor for pushes. Click on Settings > Webhooks:

Once we click Add Webhook, the Github Server will send a request to our Jenkins server and if successful it will show a green check mark next to the URL when we go back to the Webhooks section of our repository:

Now let’s see how to use this webhook in Jenkins:

Select "FreeEstyle" project.
Click on the 'Source Code Management' tab.
Click on Git and paste the GitHub repository URL in the 'Repository URL' field.
Note that we do not need to put in the "Credentials" here.
On the 'GitHub hook trigger for GITScm polling'.
Click on "Save."

Now, whenever a new push to Github, our Jenkins task will get triggered:

Of course, we can use the Webhook with other type of projects such as Pipeline.

Here, we're using the "feature" branch of the same repo.

(Optional) A sample Jenkinsfile with Maven

Just a sample Jenkinsfile:

pipeline {
    agent {
        docker {
            image 'maven:3-alpine'
            args '-v /root/.m2:/root/.m2'
        }
    }
    stages {
        stage('Build') {
            steps {
                sh 'mvn -B -DskipTests clean package'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/*.xml'
                }
            }
        }
        stage('Deliver') {
            steps {
                sh './jenkins/scripts/deliver.sh'
            }
        }
    }
}