Test Coverage

Enabling this Analyzer will show you coverage metrics like Line Coverage, Branch Coverage, Condition Coverage, and, Composite Coverage. The Analyzer also looks at lines of code that were never executed by any tests and raises actionable issues for them.

This section covers steps to get started with the test-coverage Analyzer. Please make sure to read the general configuration guide first.

​

Configuration - .deepsource.toml

To enable the Test coverage Analyzer, make sure you have it enabled in your .deepsource.toml configuration file. This can be done by adding the following entry:

​

Setup Test Coverage

The analysis begins as soon as the Analyzer recieves a valid coverage report. To do this, you can hook up the DeepSource CLI with your test CI and send the coverage report as soon as the tests finishes.

​

Set up DEEPSOURCE_DSN environment variable

DSN is used to associate the coverage artifact to the repository. This needs to be configured before sending a coverage artifact for analysis.

To look up the DSN:

• Go to the Settings page of the repository dashboard in DeepSource
• Go to the Code Coverage Section
• Click on Copy button to copy your DSN.

Once you get the DSN, please make sure you store it as an environment variable named DEEPSOURCE_DSN in your CI’s environment.

​

Installing DeepSource CLI in your CI enviroment

This command will create a bin directory in the command’s current working directory($PWD/$0). deepsource executable will be present inside bin. We would use this executable for our CLI to DeepSource from now.

​

Reporting coverage artifact using the cli

This is the final step. Once you have the Test coverage reports generated. You can send it to DeepSource using the CLI.

Here’s how the command looks like:

In the above command, <key> is the language for which you’d be sending the coverage report. Here’s the whitelist of the keys and their respective formats we support.

Please refer ‘report’ command of CLI documentation if you want to read more on the command usage.

​

Submitting Multiple coverage reports

DeepSource supports merging coverage reports implicitly. If you have multiple CI pipelines generating partial coverage reports, send them as soon as they are generated under the same key name. DeepSource will combine all of them to prepare a final result. For example, if two CI pipelines test platform-specific parts of a module, you can report both the artifacts, and DeepSource will implicitly combine the results of the reports.

You’ll notice a newly updated check every time you submit a new artifact. There’s no time limit for sending multiple coverage reports.

​

Examples

​

JavaScript

​

Jest

Currently, only the Cobertura XML format is supported by DeepSource.

Here are the steps to report JavaScript coverage data.

​

Go

​

go test

The analyzer supports coverage profile (report) for all three modes — set, atomic, and count generated by the go test.

This is how you can generate a coverage profile and report it to the analyzer:

Note: Tests for different Go packages

If you are running tests and generating coverage reports for different packages separately, you need to combine all generated coverage reports into a single file and submit that to the test-coverage analyzer.

This is how you can combine the coverage artifacts before reporting them:

​

Python

​

unittests

​

pytest

Reference: https://pypi.org/project/pytest-cov/

​

nose2

Reference: https://docs.nose2.io/en/latest/plugins/coverage.html

​

NOTE: Usage with tox

If you are using tox.readthedocs.io, make sure you omit .tox/* and env/* in your .coveragerc file because when running tests with tox, the coverage report will contain test report files other than your source (e.g site-packages, etc).

Here’s an example of a .coveragerc file:

With multiple python versions, combine the coverage data files before reporting.

Example tox.ini file:

​

Ruby

​

SimpleCov

First, install simplecov if it is not already installed.

Then, follow the steps below to generate a test coverage report.

1. Add the following lines to the spec_helper.rb file inside the tests folder of your project:

2. Add --require spec_helper.rb to the .rspec file.
3. Run rspec using bundle exec rake rspec to generate a coverage report.
4. The coverage report will be available inside the coverage folder.

Once you have a coverage report, you can upload it to DeepSource using the following commands:

​

Java

The test coverage analyzer supports test coverage metrics for Jacoco and Clover XML reports.

​

Jacoco

Setting up test coverage differs with each type of build system (Maven, Gradle, etc.). Here’s an example of the configuration needed to run Jacoco on a maven repo:

Once you’ve added Jacoco to your project’s pom.xml file, you should be able to run tests and generate the coverage report. The default location of the coverage report is target/site/jacoco/jacoco.xml.

In case your project has multiple modules, you will need to use the jacoco:report-aggregate goal to merge all reports together.

After you have the XML test report, you can upload it to DeepSource using the cli:

You should be able to proceed similarly with other build systems such as Ant or Gradle.

Reference: Jacoco documentation

​

Clover

DeepSource also supports reports generated using Atlassian’s Clover coverage framework. Here’s an example of using it with Maven:

Once you have added these elements to your project’s pom.xml file, you will be able to instrument and run tests:

This will run tests and in the case of a multimodule project, aggregate the results into a single clover report.

The default output directory for the report is target/site/clover/clover.xml.

After you have the XML test report, you can upload it to DeepSource using the cli:

You should be able to proceed similarly with other build systems such as Ant or Gradle.

Reference: Clover documentation

​

Kotlin

The test coverage analyzer supports test coverage metrics for Jacoco XML reports.

​

Jacoco

Setting up test coverage differs with each type of build system (Maven, Gradle, etc.). Here’s an example of the configuration needed to run Jacoco on a maven repo:

Once you’ve added Jacoco to your project’s pom.xml file, you should be able to run tests and generate the coverage report. The default location of the coverage report is target/site/jacoco/jacoco.xml.

After you have the XML test report, you can upload it to DeepSource using the cli:

In case your project has multiple modules, you will need to use the jacoco:report-aggregate goal to merge all reports together.

​

Kover

If you have a gradle project, you can set up kover for report generation as well.

Add the following in your top-level build file:

Once you’ve applied the Kover Gradle plugin, Kover tasks for report generation and verification will be created. To generate XML reports, you can use the following command:

./gradlew koverXmlReport

koverXmlReport will build, execute tests and then write out an XML report in the specified report location. The default location for Kover reports is:build/reports/kover/xml/report.xml.

Note that kover will automatically run your full test suite by default. If you instead want to execute tests separately, make sure to include -x test in gradle’s arguments.

You can then use the DeepSource CLI to upload this report:

​

Scala

​

Jacoco

Add sbt-jacoco to your project by adding the following line to your project/plugins.sbt -

addSbtPlugin("com.github.sbt" % "sbt-jacoco" % "<version>")

Then run sbt jacoco to generate the coverage report. By default, you’ll find the report in /target/scala-{version}/jacoco/report.

Note: To tweak Jacoco and its coverage behavior, you’ll have to make changes to your build.sbt. Refer Jacoco’s documentation for more information on this.

Once you have a coverage report, you can upload it to DeepSource using the following commands:

Note: Since we support Cobertura for other languages as well, you should be able to follow the above similar steps for generating and uploading your Cobertura report.

​

PHP

​

PHPUnit

Currently, only the Cobertura XML format is supported by DeepSource.

Here are the steps to report PHP coverage data.

​

Rust

​

C & C++

​

GCC & LCOV

​

C#

​

Via dotnet test

​

Swift

Currently, we support the LCOV format.

Here is an example of how to generate a coverage report in LCOV format using swift test and llvm-cov.

​

Reporting coverage from tests running in Docker container

Running tests inside a Docker container is a widely used practice in the software development community since it provides a consistent environment to run the application.

But, at the same time it also adds isolation and therefore, the test coverage reports generated inside the container are not accessible to the outside environment i.e. the CI systems on which the testing pipeline is running.

However, the following two methods can be used to report the test coverage data to DeepSource.

​

Inside the Docker container

For reporting test coverage to DeepSource from inside the container which runs tests, just pass some environment variables to the container using the --env/-e flag.

​

Outside the Docker container

The test coverage report can also be reported by copying it from the Docker container in which tests are run to a shared directory which the host can also access.

In the Dockerfile of the container which runs test, make sure that the generated test coverage report is moved to the shared directory.

​

Automating Test Coverage Tracking with CI

If you’re using a CI to run your tests, the recommended way to use the Test Coverage Analyzer is by automating the coverage reporting from your CI. This should be done in the scripts that are running your tests.

These are the things that you’d need to take care of:

1. Enable the test-coverage Analyzer in your repository’s .deepsource.toml file.
2. Make sure that your CI is checking out the same commit and not making a merge commit. Failing to do this would cause the coverage report to be associated with the merge commit, and DeepSource would never pick it up for a run.
3. Install the DeepSource CLI. This would be needed to report the coverage artifact.
4. DeepSource DSN for the repository. This is needed to identify the repository for which the coverage report is being sent. The CLI looks for the DSN

The sections below contain boilerplate config for different CI providers. Please refer to this section to learn more about the CLI’s report command used in the examples.

​

With Travis CI

1. On Travis CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN environment variable with the DSN copied above as its value.

2. Add this to .travis.yml:

​

With Circle CI

1. On Circle CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN environment variable with the DSN copied above as its value.

2. Add the following step in .circleci/config.yml:

​

With GitHub Actions

1. On GitHub, navigate to the main page of the repository. Under your repository name, click “Settings”. In the left sidebar, click Secrets.

   • Type DEEPSOURCE_DSN in the “Name” input box.
   • Add the value copied above.

2. When you checkout code, ensure that you use pull request HEAD commit instead of merge commit:

   GitHub actions make a merge commit during checkout unless it is asked not to do it. If you’re using GitHub actions, please make sure you make the following change in your checkout action.

3. Add the following step in .github/workflows/main.yml:

​

With GitLab CI

1. Navigate to the project page of the repository on GitLab. Under project settings, in the sidebar, click on “CI/CD”. Expand the variable section, and add the following:

   • Type: “Variable”`
   • Key: DEEPSOURCE_DSN
   • Value: The DSN value copied above
   • State: Protected (Yes)
   • Masked: Yes
   • Scope: All Environments

2. Add the following under the test job in .gitlab-ci.yml:

​

With Heroku CI

1. Navigate to the app’s Settings tab in the Heroku Dashboard and then add the Config Variables:

   • KEY: DEEPSOURCE_DSN
   • VALUE: The DSN value copied above

2. Run the following commands:

​

With Azure Pipelines

1. Set a secret variable for the repository’s DSN to be used in the pipeline.

   • Name: DEEPSOURCE_DSN
   • Value: The DSN value for your repository on DeepSource

2. Add the following script in the pipeline(s) running tests for the repository:

env:
GIT_COMMIT_SHA: $(System.PullRequest.SourceCommitId)