Setting up code coverage reports in Azure DevOps pipeline
Code coverage in .NET Core projects with Azure DevOps
I while ago I wrote and article about Publishing .NET Core code analysis to SonarCloud from Azure build pipeline. Although SonarCloud is a great platform for analyzing your code coverage and dry code analysis, it can add additional cost to your project. SonarCloud is only free for open-source projects, but if you have your private repository than you will have to pay for commercial tier.
Azure DevOps build pipeline has a code coverage option as well, but in order to have it work with .NET Core you have to setup the reporting yourself. With SonarCloud you only need to publish test results and it will do the reporting for you. With Azure DevOps you need to do this step yourself, meaning that apart from generating the test results with the unit tests step (dotnet test) you need to generate the code coverage report (HTML) manually in the pipeline itself.
This is not so hard considering Visual Studio Test Platform has Coverlet format data collector already integrated in it, so it comes pretty much out of the box. It is pretty much a matter of command line argument to get Coverlet format reports.
It is important that you reference coverlet.collector nuget package in your unit tests project. It is only required to reference it in unit tests project.
<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <TargetFramework>netcoreapp3.1</TargetFramework> <IsPackable>false</IsPackable> </PropertyGroup> <ItemGroup> <PackageReference Include="AutoFixture" Version="4.11.0" /> <PackageReference Include="Microsoft.CodeCoverage" Version="16.6.1" /> <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" /> <PackageReference Include="Moq" Version="4.14.1" /> <PackageReference Include="Newtonsoft.Json" Version="12.0.3" /> <PackageReference Include="xunit" Version="2.4.0" /> <PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" /> <PackageReference Include="coverlet.collector" Version="1.2.0" /> </ItemGroup> </Project>
Second part, generating HTML report from the Coverlet data can be easily done with .NET Core Report Generator global tool. It is already available from the NuGet.org package repository, so all you need to do is to install it before publishing the coverage results in the pipeline.
I have prepared a sample pipeline for this article which basically executes the following tasks:
- restores project packages (dotnet restore)
- build the target project (dotnet build)
- executes the unit tests related to the project previously built (dotnet test) and generate Coverlet format results
- install dotnet-reportgenerator-globaltool dotnet tool with Command Line task (dotnet tool install)
- execute reportgenerator tool via Command Line task which is previously installed targeting previously generated test results
- publish code coverage results to Azure DevOps
To make things as simple as possible, I will only focus on unit tests project and omit the actual project
Let's walk through each step and see how are they configured in Azure DevOps pipeline definition YAML file.
Restore project packages (dotnet restore)
First step is to restore nuget packages for the projects before the build. This will pull all packages to local folder of the build agent.
Running dotnet run command implicitly calls dotnet restore so you can omit this step. For more clarity what is going on in the flow I prefer to have both steps invoked explicitly
- task: DotNetCoreCLI@2 displayName: 'dotnet restore' inputs: command: restore projects: '**/MyProject.UnitTests.csproj' includeNuGetOrg: true
Build projects (dotnet build)
Second step is project build. This will compile project file (.csproj) to assemblies (.dll) which will be later used as an executable.
- task: DotNetCoreCLI@2 displayName: 'dotnet build' inputs: command: build projects: '**/MyProject.UnitTests.csproj'
Run unit tests
So far all pipeline tasks were pretty much straight forward, but with unit tests we need to instruct dotnet CLI to collect test results to cobertura format. This is done with a command line argument.
- task: DotNetCoreCLI@2 displayName: 'dotnet test' inputs: command: test projects: '**/MyProject.UnitTests.csproj' arguments: '--configuration $(BuildConfiguration) --collect "XPlat Code coverage" -- RunConfiguration.DisableAppDomain=true' testRunTitle: MyProject.UnitTests
More about fine tuning and options for coverlet with .NET you can find on coverlet project page https://discoverdot.net/projects/coverlet
Install report generator tool
To install report generator tool I used Command Line task, as I had issues configuring .NET Core task to use custom command. Down the line, it translates to a same command which executes on the build agent host.
- script: 'dotnet tool install --global dotnet-reportgenerator-globaltool --version 4.5.8' displayName: 'Install ReportGenerator tool'
Generate reports with reportgenerator tool
Report generating step is also a Command Line task in the build pipeline but it is a separate task rather than additional line in global tool install task. The reason for that is that once installed, the tool will be available in the new session of bash (on Linux) or cmd (on Windows).
If you bundle lines together, the line that invokes reportgenerator and generates report will fail. For that reason this action is in a separate task which will create a new command line session and therefore have previously installed tool available.
- script: 'reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/coverlet/reports -reporttypes:"Cobertura"' displayName: 'Create reports'
A configured only minimal set of options, but you can add bunch of customizations to your reports by including supported arguments. More details and documentation related to reportgenerator tool you can find on the tool's official page on Github https://github.com/danielpalme/ReportGenerator.
Publish code coverage to Azure DevOps
This is a final step and basically represents uploading of all HTML pages to Azure DevOps pipeline, so that they are visible from the Azure DevOps UI.
- task: PublishCodeCoverageResults@1 displayName: 'Publish code coverage' inputs: codeCoverageTool: Cobertura summaryFileLocation: '$(Build.SourcesDirectory)/coverlet/reports/Cobertura.xml'
For this purpose I use Publish Code Coverage Results task with really minimal configuration of only mandatory fields.
Once the build pipeline is executed, results are visible in Code Coverage Report tab of the build. For some reason Azure DevOps has several places in the build overview that are mentioning code coverage but they are actually available only from one which is marked on this image.
As I mentioned, the report is generated with minimal options and configurations involved, to get more out of your code analysis checkout documentation and arguments reports generating tool supports from it's official page on Github https://github.com/danielpalme/ReportGenerator.
Purpose of the code contained in snippets or available for download in this article is solely for learning and demo purposes. Author will not be held responsible for any failure or damages caused due to any other usage.