Code coverage and Metrics

Jenkins

We can also publish a large number of useful reports using Jenkins's plugins. Plugins are one of Jenkins's strong points - there are plugins for doing just about anything, from integrating new build tools or version control systems to notification mechanisms and reporting. In addition, Jenkins plugins are very easy to install and integrate smoothly into the existing Jenkins architecture.

To see how the plugins work, we are going to integrate code coverage metrics using the Cobertura plugin (1.9.6). Code coverage is an indication of how much of our application code is actually executed during our tests - it can be a useful tool in particular for finding areas of code that have not been tested by our test suites. It can also give some indication as to how well a team is applying good testing practices such as Test-Driven Development or Behavior-Driven Development.

Cobertura1 is an open source code coverage tool that works well with both Maven and Jenkins. Our Maven demonstration project is already configured to record code coverage metrics, so all we need to do is to install the Jenkins Cobertura plugin and generate the code coverage metrics for Jenkins to record and display.

Let's go to the Available tab and scroll down until we find the Cobertura Plugin entry in the Build Reports section. Click on the checkbox and then click on the Install button at the bottom of the screen:

This will download and install the plugin for us. Once it is done, we will need to restart our Jenkins instance:

$ sudo service jenkins restart
 * Restarting Jenkins Continuous Integration Server jenkins       [ OK ]

When we have restarted Jenkins, go back to the Manage Plugins screen and click on the Installed tab - there should now be a Cobertura Plugin entry in the list of installed plugins on this page.

1. Install the cobertura plugin (via Manage Jenkins -> Manage Plugins) (Done in previous section).
2. Configure project's build script to generate cobertura XML reports.
3. Enable the "Publish Cobertura Coverage Report" publisher.
4. Specify the directory where the coverage.xml report is generated.
5. (Optional) Configure the coverage metric targets to reflect goals.

Now that the plugin was successfully installed, let's go to the configuration page for the gameoflife-default build job.

To set up code coverage metrics in our project, we need to do two things:

1. First we need to generate the Cobertura coverage data in an XML form that Jenkins can use.
2. Then, we need to configure Jenkins to display the coverage reports.

Our Game of Life project already has been configured to generate XML code coverage reports if we ask it. All we need to do is to run mvn cobertura:cobertura to generate the reports in XML form. Cobertura can also generate HTML reports, but in our case we will be letting Jenkins take care of the reporting.

We will just add the cobertura:cobertura goal to the second build step.

Samples of Cobertura goals:

1. cobertura:check Check the Last Instrumentation Results.
2. cobertura:clean Clean up rogue files that cobertura maven plugin is tracking.
3. cobertura:dump-datafile Cobertura Datafile Dump Mojo.
4. cobertura:instrument Instrument the compiled classes.
5. cobertura:cobertura Instruments, Tests, and Generates a Cobertura Report.

Next, we need to tell Jenkins to keep track of our code coverage metrics. Scroll down to the "Post-build Actions" section. We should see a new checkbox labeled "Publish Cobertura Coverage Reports":

Jenkins will often add UI elements like this when we install a new plugin. When we tick this box, Jenkins will display the configuration options for the Cobertura plugin that we installed earlier.

Like most of the code-quality related plugins in Jenkins, the Cobertura plugin lets us fine-tune not only the way Jenkins displays the report data, but also how it interprets the data. In the "Coverage Metrics Targets" section, we can define what we consider to be the minimum acceptable levels of code coverage.

We have configured Jenkins to list any builds with less than 50% test coverage as "unstable" indicated by a yellow ball, and notify the team accordingly.

This fine-tuning often comes in handy in real-world builds. For example, we may want to impose a special code coverage constraint in release builds, to ensure high code coverage in release versions. Another strategy that can be useful for legacy projects is to gradually increase the minimum tolerated code coverage level over time. This way we can avoid having to retro-fit unit tests on legacy code just to raise the code coverage, but we do encourage all new code and bug fixes to be well tested.

This build failed, and I'll come back to this later.

[ERROR] Nov 12, 2014 9:06:52 PM net.sourceforge.cobertura.coveragedata.CoverageDataFileHandler loadCoverageData
INFO: Cobertura: Loaded information on 2 classes.

[INFO] Cobertura Report generation was successful.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] gameoflife ........................................ SUCCESS [ 24.183 s]
[INFO] gameoflife-build .................................. SUCCESS [  3.759 s]
[INFO] gameoflife-core ................................... SUCCESS [ 13.105 s]
[INFO] gameoflife-web .................................... SUCCESS [ 10.941 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59.236 s
[INFO] Finished at: 2014-11-12T21:06:52-08:00
[INFO] Final Memory: 30M/279M
[INFO] ------------------------------------------------------------------------
[Cobertura] Publishing Cobertura coverage report...
[Cobertura] No coverage results were found using the pattern ' **/target/site/cobertura/coverage.xml' relative to '/var/lib/jenkins/workspace/game-of-life'.  Did you enter a pattern relative to the correct directory?  Did you generate the XML report(s) for Cobertura?
Build step 'Publish Cobertura Coverage Report' changed build result to FAILURE
Finished: FAILURE

Jenkins