SonarQube Plugin for Existing NUnit Reports (Work in progress)

Header Photo Credit: Lorenzo Cafaro (Creative Commons Zero License)

Posted on Jan 24, 2014
 || Last Updated: Feb 01, 2014

UPDATE 2/1/2014: After I wrote this initial posting, the team at SonarSource who oversee SonarQube and the .NET plugins replied to me saying they have changed their mind on the need for this plugin, and on Unit Test reporting in general. Going forward, SonarQube will not be reporting unit test results, focusing only on coverage details. As such, this plugin is effectively dead. I do not plan on continuing any work on it at this time.

I've made some decent progress with a new SonarQube plugin, intended to be part of the community "C# Ecosystem" plugin package, which I'm calling the "sonar.dotnet.tests" plugin. The intent of this plugin is to support direct parsing of unit testing/coverage tools' reports instead of relying on Gallio, as the current ecosystem plugins do. The first (and currently only) supported framework is NUnit, allowing you to directly import NUnit test results into SonarQube without going through Gallio (see SONARDOTNT-372).

I posted this email to the SonarQube developers mailing list, but thought I'd also post the information here, so that it's easier to find via search engines and so that people can leave comments without having to subscribe to the mailing list.

If you're interested in taking a look at my work in-progress, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository. Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch. There is not currently a publicly hosted build of the plugin, so if you want to try it out, you'll need to build it (and the other dotnet plugins in the same repo) yourself.

Some general notes:

  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only reuseReport and skip modes.

Implementation details:

  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property (sonar.csharp.squid.includeTestSources) to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is false to mirror prior behavior). This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this: /!\\ Do not use for the moment!For the moment, method key ends with ':XXXX', where 'XXXX' is the line number, so this API does not work.TODO: Need to work on that.
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature. I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here". I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:

For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will be spending my "side project time" working with the That Conference organizers team for That Conference 2014. I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

This entry was posted in Tools , Code Analysis , SonarQube .