Comparison of code coverage tools

免責事項

We've tried to do our best to prepare non-biased, based on features, comparison of various code coverage tools available on the market in order to help in evaluation process. Information gathered here is based on the official tools' documentation as well as on documentation of tools' integrations. In case you'd find some information presented here inaccurate or outdated or if you think that new tool or feature shall be added to the comparison, please do not hesitate to add a comment to the page - we will review it and update the page accordingly.

機能	Atlassian Clover	Cobertura	JaCoCo	Code Cover	PIT
ホームページ
	atlassian.com/software/clover	cobertura.github.io/cobertura	eclemma.org/jacoco	codecover.org	pitest.org
Instrumentation
ソースファイル
Class files		off-line instrumentation	off-line and on-the-fly* instrumentation*
Coverage metrics
Method
Statement ⁽¹⁾
Line ⁽¹⁾
ブランチ
MC/DC
手順説明			explanation
Global coverage
Per-test coverage		sonar	JMX / sonar		lists tests per file
Mutation coverage
Source code metrics
Available metrics	20+ metrics, also custom ones	cyclomatic complexity	cyclomatic complexity
レポートタイプ
HTML	more details
PDF
XML
JSON
Text		via cobertura:check
CSV
Data management and report filtering
Merging of coverage databases	clover2:merge		via <jacoco:merge>
Historical reporting	more details	via sonar	via sonar
Selecting scope of code coverage	file patterns, class patterns, method patterns (entire signature), code block type, statement's regular expression, code complexity, CLOVER:OFF/ON code comments	file patterns, code annotations	class patterns	file patterns	package patterns
Cross-report linking			via <structure> element
Supported languages
Java
groovy	Clover recognizes Groovy-specific language constructs, such as "?.", "?:" operators, properties, methods with default arguments. It also handles Grails-specific AST transformations such as controller classes. Thanks to this, you can see more accurate data in reports (compared to pure byte-code instrumentation tools).
その他	⁽²⁾	instrumentation is class-based so theoretically any JVM language is supported, but it may lack good reporting (esp. for language-specific constructs); it may also have problems with synthetic methods etc.
Supported JDK
	1.6-1.8 (JRE/JDK) 1.3-1.8 (for "-source" level setting)	1.5 〜 1.7	1.5 〜 1.8	1.5 〜 1.7	1.5 〜 1.8
Supported test frameworks
JUnit		⁽⁴⁾	⁽⁵⁾
TestNG		⁽⁴⁾	⁽⁵⁾
Spock	See Clover 3.3 Release Notes	⁽⁴⁾	⁽⁴⁾
その他	It's possible to define custom patterns for tests, based on file names, class signatures, method signatures (including annotations and javadoc tags). More details.	⁽⁴⁾	⁽⁴⁾	It's possible to define test boundaries by special instructions put in code comments
IDE integrations
IntelliJ IDEA
Eclipse		eCobertura
NetBeans
Build tools integrations
command line
ant
Maven
Grails		code-coverage
Gradle	⁽³⁾	gradle-cobertura-plugin
SBT	sbt-clover	cobertura4sbt
CI servers integrations
Bamboo
Hudson
Jenkins
TeamCity
その他の連携
Sonar	⁽³⁾
Jira
Development activity
Last release	actively developed, about 7 releases / year	minor activity, last release - 2013	actively developed, few releases / year	minor activity, last release - 2011	actively developed, few releases / year
テクニカルサポート
	Atlassian Support, 24h response	open source community	open source community	open source community	open source community
Subjective summary
優位性	Clover has great and highly configurable HTML reports (showing not only code coverage but also top risks etc), per-test code coverage and test optimization, distributed per-test coverage and many tool integrations; it is being actively developed and supported.	Easy to use thanks to off-line byte code instrumentation. You can measure coverage without having the source code. It has very nice and easy to navigate HTML report (example).	Very easy to integrate thanks to the on-the-fly byte code instrumentation. You can measure coverage without having the source code. It has nice HTML report (example).	It has the most detailed code coverage metric (MC/DC), which may be useful for critical systems (medical, aeronautical etc). The Eclipse plug-in comes also with a cool Boolean Expression Analyzer view and a Test Correlation matrix. It has also an interesting feature to start/stop test case via JMX, which can be useful for manual testing.	PIT is a tool for mutation coverage, which means it will not only measure line coverage of your code but will also perform mutations in application logic in order to check how well written your tests are.
欠点	Due to a fact that Clover is based on source code instrumentation, integration requires a build - it's necessary to recompile code with Clover. Most Clover's integrations have an automatic integration feature, but in some cases you may need to add Clover JAR to a class path or set some Clover options.	Classes must be compiled with debug option.	Classes must be compiled with debug option.	Last release has been performed 3 years ago. The HTML report generated is quite fragmented - source code is shown separately for every method.

Statement / line metrics are very similar in terms of their granularity.
Scala support is one of the top-voted features. See Clover+Road+Map.
Plug-in not supported by Atlassian.
While it's possible to instrument test classes and run test frameworks with Cobertura and JaCoCo, there is no built-in, dedicated support for these frameworks. Which means that standard HTML report shows neither test results nor per-test coverage. There is also no per-test data in Eclipse IDE (EclEmma, based on JaCoCo).
Sonar can generate separate coverage data sets for every test case and show them in a combined report. See http://deors.wordpress.com/2014/07/04/individual-test-coverage-sonarqube-jacoco/

ページツリー

Comparison of code coverage tools