Integrating Karma 0.8 tests in Maven with Sonar(Cube) test coverage

NOTE: this blog post was written for version 0.8 of the Karma test runner.
An updated blog post for the new Karma 0.10 can be found here.

For my current project we are using Maven to build our AngularJS application.
Furthermore we use Sonar (recently renamed to SonarCube) to monitor our code standards / best practices and unit test coverage.

In this blog post we describe how to integrate the Karma (Testacular) test runner with Maven and how to add your AngularJS (or any JavaScript) application to SonarQube.

Basic integration of the Karma test runner in Maven

When I initially integrated the Karma test runner with Maven I used the Maven AntRun plugin because there was no Karma plugin for Maven at that moment.
Luckily integrating Karma into Maven becomes much easier using the official maven-karma-plugin:

In the <configuration> section we overrule the following settings from the “karma.conf.js” Karma configuration file:

  1. Instead of using the browsers specified in the Karma configuration file we want to use the “headless” browser “PhantomJS” in order not to spawn any browser windows during our Maven build.
  2. To allow the Karma progress to be properly send its output to a log file we use “dots” for progress reporting instead of the standard percentage that’s being updated.
  3. Since colors are also not possible in a log file we also disable the use of colors.
  4. Finally we explicitly turn off “autowatch” because we are executing a “single run” in Karma (which is the default behavior when using the Karma Maven plugin) and want to prevent the following unwanted message:
    config - autoWatch set to false, because of singleRun

Generating HTML unit test coverage reports

To generate HTML unit test coverage reports while running the Karma test runner we will have to modify the “karma.conf.js” Karma configuration file:

  1. To pre-process the source files to allow test coverage reporting;
    this process of altering the sources is also known as instrumentation.
  2. To contain an additional “coverageReporter” section to specify that we want HTML reporting and the location of the generated reports

NOTE: this blog post uses the (old) pre-0.10 configuration file format of the Karma test runner.
However when using version 0.10 all snippets (and sample application) should still be working.

In order to generate the unit test coverage you will have to append “,coverage” to the <reporters> in the Maven POM:

The pre-processing of JavaScript source files dramatically alters the code, causing the line / column numbers of the altered code to no longer match the original ones.
Therefore the stack traces of exceptions that occur while testing will also use those altered line / column numbers.

Instead of always pre-processing the sources we could disable it at default by commenting it out:

Instead we use the Maven AntRun Plugin to generate a additional file “target/karma-coverage-html.conf.js” with those lines uncommented:

After running (at least) a mvn initialize a separate “target/karma-coverage-html.conf.js” file is generated to be used for HTML test coverage reporting while using Karma (outside the Maven build) on the command-line or from within Webstorm / IntelliJ IDEA.

The generated Karma configuration file differs from the original Karma configuration file in that:

  • the “basePath = '';” line is altered to “basePath = '..';” to let Karma know that all paths are relative to “..” (since the generated “karma-coverage-html.conf.js” is located in “target” whereas all paths in the Karma configuration file are still relative to the directory in which “karma.conf.js” is located)
  • each earlier commented-out “ // '...': 'coverage'” line is stripped from its comment (therefore enabling the pre-processing of the sources)

In order to use the generated “target/karma-coverage-html.conf.js” Karma configuration file you will need to add a <configFile> element to the configuration of the Maven Karma Plugin:

NOTE: Later in this blog post we will modify our Maven POM to generate an additional Karma configuration file for LCOV test coverage reporting and configure the <configFile> element of the configuration of the Maven Karma Plugin to use it.

Basic integration with SonarCube (no unit test coverage yet)

For a basic integration of your JavaScript code in SonarCube you will have to specify:

  1. That this particular Maven module is using the JavaScript language (instead of the default Java)
  2. That all sources in “lib/**” (containing JavaScript libraries) should be ignored.
  3. That SonarCube should reuse existing reports for dynamic analysis
  4. That information about the executed unit tests can found in “target/surefire-reports”
  5. The source directory containing our JavaScript source-code.
  6. The “test” source directory containing our JavaScript (i.e. Jasmine) unit tests.
  7. That the Karma test runner should create a JUnit style XML report
    (containing information about the executed JavaScript unit tests)

Additionally, in order for Karma “junit” reporter to work we will have to specify in the “karma.conf.js” file where the JUnit XML report should be generated:

NOTE: as it turns out the JavaScript plugin of SonarCube (as of version 1.5) no longer supports importing “unit test execution reports”. More information can be found here (see answer of “Ronald”) and here.

Integrating unit test coverage with SonarCube

In order to integrate unit test coverage with SonarCube we will need to used LCOV reporting (instead of HTML).

For this we need to add an additional <execution> section to our Maven AntRun Plugin configuration to generate an additional file “target/karma-coverage-lcov.conf.js” in which we also modify the Karma configuration to (only) output LCOV test coverage reporting:

The generated “target/karma-coverage-lcov.conf.js” file differs from the original Karma configuration file in that:

  • the “basePath = '';” line is altered to “basePath = '..';
  • each earlier commented-out “ // '...': 'coverage'” line is stripped from its comment
  • the “type : 'html',” line is altered to “type : 'lcovonly',” to let Karma generate LCOV coverage reporting (for consumption by SonarCube) instead of HTML coverage reporting.

In order to use the generated Karma configuration file you will need to modify the (earlier added) <configFile> element in the configuration of the Maven Karma Plugin:

Then we need to tell SonarCube where to find the LCOV unit test coverage report:

NOTE: later in this blog post we present a more future proof to specify the location of the LCOV file that doesn’t depend on the browser version in its path (like in this case “1.8” of “PhantomJS” on “Windows”).

Configuring <sonar.javascript.lcov.reportPath> isn’t enough to make SonarCube properly process the LCOV file.

To successfully let SonarCube process the generated LCOV file we will add an additional <execution> section to the configuration of the Maven AntRun plugin that:

  1. Copies the generated “lcov.info” file from the “target/karma-coverage/PhantomJS*/” directory to the “target” directory;
    this allows us to change the <sonar.javascript.lcov.reportPath> property to target/lcov.info

NOTE: for correct LCOV processing, ensure the latest SonarCube JavaScript plugin (1.4) is installed.

Sample application

To demonstrate the integration of the Karma test runner in Maven and SonarCube I forked AngularJS sample application called “ng-directive-testing” in GitHub.

After running a “mvn clean install sonar:sonar” (either from command-line or through Jenkins) you will see the following dashboard in SonarCube:
sonar-dashboard

References

7 thoughts on “Integrating Karma 0.8 tests in Maven with Sonar(Cube) test coverage

  1. Note that in the “Generating HTML unit test coverage reports” section, your item “1” and “2” are reversed in the code sample. The “(1)” should be “(2)” and the “(2)” should be “(1)”.

    At the point in this when you said “Instrumenting your JavaScript …”, that’s when this starts to get bizarre. I understand it’s probably necessary, but I wish you could elaborate on exactly why you’re doing this, and drill down into more detail on the transformation you’re making to the conf file.

    • Thank you for your reply. It’s always great to receive feedback in order to improve / correct an article.

      I just updated the blog post to correct the “Generating HTML unit test coverage reports” section.
      Additionally I’ve rewritten some sentences (related to the pre-processing of sources) and added additional explanations about what is modified in the “target/karma-coverage-html.conf.js” and ““target/karma-coverage-lcov.conf.js” files.

  2. Hi there. I am just starting out with “pure” javascript projects. I noticed that you got coverage data into sonar, but the information about the unit tests themselves are not there (ie. which tests has run and how much time they took etc..). Have you been able to fix that? I am unable to specify the reportspath correctly so SonarQube will not pick up my TEST-karma.xml file.

    • It has been quite some while since I actually tested SonarCube’s support for JUnit XML reporting files.
      I guess that somehow the paths in the generated XML file is somehow not to SonarCube’s liking. Have you tried using the recent 0.10 version of the Karma test runner (and my blog post about it)?

      • After doing some initial investigation it turns out that support for importing “unit test execution reports” has been dropped from the version 1.5 of the SonarCube JavaScript plugin. Additionally the “mavenized-karma-0_8-ng-directive-testing” sample application (as it turns out) wasn’t not configured at all (sorry for that) to export an JUnit XML report and to import it into SonarCube.

        I will update the blog posts (also the one for version 0.10 of karma) with a comment about the fact that JUnit XML reports can no longer be imported in SonarCube.

Leave a Reply

Your email address will not be published. Required fields are marked *