Enabling Code coverage will show you coverage metrics like Line Coverage, Branch Coverage, Condition Coverage, and, Composite Coverage. The Analyzer also looks at lines of code that were never executed by any tests and raises actionable issues for them.
This section covers steps to get started with the test-coverage
Analyzer. Please make sure to read the general configuration guide first.
.deepsource.toml
To enable the Test coverage Analyzer, make sure you have it enabled in your .deepsource.toml
configuration file. This can be done by adding the following entry:
The analysis begins as soon as the Analyzer receives a valid coverage report. To do this, you can hook up the DeepSource CLI with your test CI and send the coverage report as soon as the tests finishes.
DEEPSOURCE_DSN
environment variableDSN is used to associate the coverage artifact to the repository. This needs to be configured before sending a coverage artifact for analysis.
To look up the DSN:
Once you get the DSN, please make sure you store it as an environment variable named DEEPSOURCE_DSN
in your CI’s environment.
This command will create a bin
directory in the command’s current working directory($PWD
/$0
). deepsource
executable will be present inside bin
. We would use this executable for our CLI to DeepSource from now.
This is the final step. Once you have the Test coverage reports generated. You can send it to DeepSource using the CLI.
Here’s how the command looks like:
In the above command, <key>
is the language for which you’d be sending the coverage report.
Please make sure you always run the above cli report
command from the root of your repo. This will make sure that the filepaths in the test reports are handled to take care directories from your CI if they are present.
Failing to do that might result in issues not getting raised for your project. You’d still be able to see the metrics.
The maximum allowed size for the coverage report file is 20 MB.
DeepSource supports merging coverage reports implicitly. If you have multiple CI pipelines generating partial coverage reports, send them as soon as they are generated under the same key name. DeepSource will combine all of them to prepare a final result. For example, if two CI pipelines test platform-specific parts of a module, you can report both the artifacts, and DeepSource will implicitly combine the results of the reports.
You’ll notice a newly updated check every time you submit a new artifact. There’s no time limit for sending multiple coverage reports.
The JaCoCo
and Clover
coverage formats report metrics for individual methods and may not contain data about the individual covered lines in the methods.
For such cases, DeepSource uses a max
operation on the reported metrics to calculate the aggregate report. So if parts of certain methods are covered in different coverage reports, the reported line coverage for that method may be lower than the actual line coverage.
For example, if one report covers the top 25% lines for a method, and another report covers the bottom 25% lines for the same method, since there is no way of knowing if both the lines in the reports are different, DeepSource will report the coverage to be 25% even though it might be 50%. This is only a limitation for JaCoCo
and Clover
coverage formats at the moment.
Running tests inside a Docker container is a widely used practice in the software development community since it provides a consistent environment to run the application.
But, at the same time it also adds isolation and therefore, the test coverage reports generated inside the container are not accessible to the outside environment i.e. the CI systems on which the testing pipeline is running.
However, the following two methods can be used to report the test coverage data to DeepSource.
For reporting test coverage to DeepSource from inside the container which runs tests, just pass some environment variables to the container using the --env/-e
flag.
The test coverage report can also be reported by copying it from the Docker container in which tests are run to a shared directory which the host can also access.
In the Dockerfile of the container which runs test, make sure that the generated test coverage report is moved to the shared directory.
If you’re using a CI to run your tests, the recommended way to use the Test Coverage Analyzer is by automating the coverage reporting from your CI. This should be done in the scripts that are running your tests.
These are the things that you’d need to take care of:
test-coverage
Analyzer in your repository’s .deepsource.toml
file.Do not add the DEEPSOURCE_DSN
variable as part of any publicly visible configuration file. It contains sensitive information.
The sections below contain boilerplate config for different CI providers. Please refer to the CLI installation section to learn more about the CLI’s report
command used in the examples.
On Travis CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN
environment variable with the DSN copied above as its value.
Add this to .travis.yml
:
On Circle CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN
environment variable with the DSN copied above as its value.
Add the following step in .circleci/config.yml
:
On GitHub, navigate to the main page of the repository. Under your repository name, click “Settings”. In the left sidebar, click Secrets.
DEEPSOURCE_DSN
in the “Name” input box.When you checkout code, ensure that you use pull request HEAD commit instead of merge commit:
GitHub actions make a merge commit during checkout unless it is asked not to do it. If you’re using GitHub actions, please make sure you make the following change in your checkout action.
Add the following step in .github/workflows/main.yml
:
You can also use OIDC to authenticate and send the coverage report to DeepSource, without using the DSN. Here are the steps to do that:
ref
input in the actions/checkout
step:Reference: GitHub Actions permissions
--use-oidc
flag to the deepsource report
command while sending the coverage report:This will use the workflow’s OIDC token to get a temporary DSN for the repository and send the coverage report to DeepSource.
Navigate to the project page of the repository on GitLab. Under project settings, in the sidebar, click on “CI/CD”. Expand the variable section, and add the following:
DEEPSOURCE_DSN
Add the following under the test job in .gitlab-ci.yml
:
Navigate to the app’s Settings tab in the Heroku Dashboard and then add the Config Variables:
DEEPSOURCE_DSN
Run the following commands:
Set a secret variable for the repository’s DSN to be used in the pipeline.
DEEPSOURCE_DSN
Add the following script in the pipeline(s) running tests for the repository:
In case of pull requests, Azure DevOps can make a merge commit that doesn’t belong to the Repo’s actual GIT tree. You can let DeepSource know about the correct commit OID of PR it should associate the coverage report to by setting the following environment variable.
Enabling Code coverage will show you coverage metrics like Line Coverage, Branch Coverage, Condition Coverage, and, Composite Coverage. The Analyzer also looks at lines of code that were never executed by any tests and raises actionable issues for them.
This section covers steps to get started with the test-coverage
Analyzer. Please make sure to read the general configuration guide first.
.deepsource.toml
To enable the Test coverage Analyzer, make sure you have it enabled in your .deepsource.toml
configuration file. This can be done by adding the following entry:
The analysis begins as soon as the Analyzer receives a valid coverage report. To do this, you can hook up the DeepSource CLI with your test CI and send the coverage report as soon as the tests finishes.
DEEPSOURCE_DSN
environment variableDSN is used to associate the coverage artifact to the repository. This needs to be configured before sending a coverage artifact for analysis.
To look up the DSN:
Once you get the DSN, please make sure you store it as an environment variable named DEEPSOURCE_DSN
in your CI’s environment.
This command will create a bin
directory in the command’s current working directory($PWD
/$0
). deepsource
executable will be present inside bin
. We would use this executable for our CLI to DeepSource from now.
This is the final step. Once you have the Test coverage reports generated. You can send it to DeepSource using the CLI.
Here’s how the command looks like:
In the above command, <key>
is the language for which you’d be sending the coverage report.
Please make sure you always run the above cli report
command from the root of your repo. This will make sure that the filepaths in the test reports are handled to take care directories from your CI if they are present.
Failing to do that might result in issues not getting raised for your project. You’d still be able to see the metrics.
The maximum allowed size for the coverage report file is 20 MB.
DeepSource supports merging coverage reports implicitly. If you have multiple CI pipelines generating partial coverage reports, send them as soon as they are generated under the same key name. DeepSource will combine all of them to prepare a final result. For example, if two CI pipelines test platform-specific parts of a module, you can report both the artifacts, and DeepSource will implicitly combine the results of the reports.
You’ll notice a newly updated check every time you submit a new artifact. There’s no time limit for sending multiple coverage reports.
The JaCoCo
and Clover
coverage formats report metrics for individual methods and may not contain data about the individual covered lines in the methods.
For such cases, DeepSource uses a max
operation on the reported metrics to calculate the aggregate report. So if parts of certain methods are covered in different coverage reports, the reported line coverage for that method may be lower than the actual line coverage.
For example, if one report covers the top 25% lines for a method, and another report covers the bottom 25% lines for the same method, since there is no way of knowing if both the lines in the reports are different, DeepSource will report the coverage to be 25% even though it might be 50%. This is only a limitation for JaCoCo
and Clover
coverage formats at the moment.
Running tests inside a Docker container is a widely used practice in the software development community since it provides a consistent environment to run the application.
But, at the same time it also adds isolation and therefore, the test coverage reports generated inside the container are not accessible to the outside environment i.e. the CI systems on which the testing pipeline is running.
However, the following two methods can be used to report the test coverage data to DeepSource.
For reporting test coverage to DeepSource from inside the container which runs tests, just pass some environment variables to the container using the --env/-e
flag.
The test coverage report can also be reported by copying it from the Docker container in which tests are run to a shared directory which the host can also access.
In the Dockerfile of the container which runs test, make sure that the generated test coverage report is moved to the shared directory.
If you’re using a CI to run your tests, the recommended way to use the Test Coverage Analyzer is by automating the coverage reporting from your CI. This should be done in the scripts that are running your tests.
These are the things that you’d need to take care of:
test-coverage
Analyzer in your repository’s .deepsource.toml
file.Do not add the DEEPSOURCE_DSN
variable as part of any publicly visible configuration file. It contains sensitive information.
The sections below contain boilerplate config for different CI providers. Please refer to the CLI installation section to learn more about the CLI’s report
command used in the examples.
On Travis CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN
environment variable with the DSN copied above as its value.
Add this to .travis.yml
:
On Circle CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN
environment variable with the DSN copied above as its value.
Add the following step in .circleci/config.yml
:
On GitHub, navigate to the main page of the repository. Under your repository name, click “Settings”. In the left sidebar, click Secrets.
DEEPSOURCE_DSN
in the “Name” input box.When you checkout code, ensure that you use pull request HEAD commit instead of merge commit:
GitHub actions make a merge commit during checkout unless it is asked not to do it. If you’re using GitHub actions, please make sure you make the following change in your checkout action.
Add the following step in .github/workflows/main.yml
:
You can also use OIDC to authenticate and send the coverage report to DeepSource, without using the DSN. Here are the steps to do that:
ref
input in the actions/checkout
step:Reference: GitHub Actions permissions
--use-oidc
flag to the deepsource report
command while sending the coverage report:This will use the workflow’s OIDC token to get a temporary DSN for the repository and send the coverage report to DeepSource.
Navigate to the project page of the repository on GitLab. Under project settings, in the sidebar, click on “CI/CD”. Expand the variable section, and add the following:
DEEPSOURCE_DSN
Add the following under the test job in .gitlab-ci.yml
:
Navigate to the app’s Settings tab in the Heroku Dashboard and then add the Config Variables:
DEEPSOURCE_DSN
Run the following commands:
Set a secret variable for the repository’s DSN to be used in the pipeline.
DEEPSOURCE_DSN
Add the following script in the pipeline(s) running tests for the repository:
In case of pull requests, Azure DevOps can make a merge commit that doesn’t belong to the Repo’s actual GIT tree. You can let DeepSource know about the correct commit OID of PR it should associate the coverage report to by setting the following environment variable.