About Code Coverage
What is Code Coverage?
Code coverage is the percentage of code which is covered by automated tests. Code coverage measurement simply determines which statements in a body of code have been executed through a test run, and which statements have not. In general, a code coverage system collects information about the running program and then combines that with source information to generate a report on the test suite's code coverage.
Code coverage is part of a feedback loop in the development process. As tests are developed, code coverage highlights aspects of the code which may not be adequately tested and which require additional testing. This loop will continue until coverage meets some specified target.
Why Measure Code Coverage?
It is well understood that unit testing improves the quality and predictability of your software releases. Do you know, however, how well your unit tests actually test your code? How many tests are enough? Do you need more tests? These are the questions code coverage measurement seeks to answer.
Coverage measurement also helps to avoid test entropy. As your code goes through multiple release cycles, there can be a tendency for unit tests to atrophy. As new code is added, it may not meet the same testing standards you put in place when the project was first released. Measuring code coverage can keep your testing up to the standards you require. You can be confident that when you go into production there will be minimal problems because you know the code not only passes its tests but that it is well tested.
In summary, we measure code coverage for the following reasons:
- To know how well our tests actually test our code
- To know whether we have enough testing in place
- To maintain the test quality over the lifecycle of a project
Code coverage is not a panacea. Coverage generally follows an 80-20 rule. Increasing coverage values becomes difficult, with new tests delivering less and less incrementally. If you follow defensive programming principles, where failure conditions are often checked at many levels in your software, some code can be very difficult to reach with practical levels of testing. Coverage measurement is not a replacement for good code review and good programming practices.
In general you should adopt a sensible coverage target and aim for even coverage across all of the modules that make up your code. Relying on a single overall coverage figure can hide large gaps in coverage.
How Code Coverage Works
There are many approaches to code coverage measurement. Broadly there are three approaches, which may be used in combination:
Source code instrumentation
This approach adds instrumentation statements to the source code and compiles the code with the normal compile tool chain to produce an instrumented assembly.
Intermediate code instrumentation
Here the compiled class files are instrumented by adding new bytecodes, and a new instrumented class is generated.
Runtime information collection
This approach collects information from the runtime environment as the code executes to determine coverage information.
Clover uses source code instrumentation, because although it requires developers to perform an instrumented build, source code instrumentation produces the most accurate coverage measurement for the least runtime performance overhead.
Be aware that while Clover is capable of instrumenting both Java and Groovy source code, the instrumentation stage occurs prior to compilation with Java and during compilation with Groovy.
As the code under test executes, code coverage systems collect information about which statements have been executed. This information is then used as the basis of reports. In addition to these basic mechanisms, coverage approaches vary on what forms of coverage information they collect. There are many forms of coverage beyond basic statement coverage including conditional coverage, method entry and path coverage.
Code Coverage with Clover
Clover is designed to measure code coverage in a way that fits seamlessly with your current development environment and practices, whatever they may be. Clover's IDE Plugins provide developers with a way to quickly measure code coverage without having to leave the IDE. Clover's Ant and Maven integrations allow coverage measurement to be performed in Automated Build and Continuous Integration systems, and reports generated to be shared by the team.
Types of Coverage measured
Clover measures three basic types of coverage analysis:
Statement
Statement coverage measures whether each statement is executed.
Branch
Branch coverage (sometimes called Decision Coverage) measures which possible branches in flow control structures are followed. Clover does this by recording if the boolean expression in the control structure evaluated to both true and false during execution.
In Groovy code, Clover also treats Elvis expressions (a :? b), safe method calls (a?.b()), safe property calls (a?.b) and safe attribute calls (a?.@b) as branches.
Method
Method coverage measures if a method was entered at all during execution.
Clover uses these measurements to produce a Total Coverage Percentage for each class, file, package and for the project as a whole. The Total Coverage Percentage allows entities to be ranked in reports. The Total Coverage Percentage (TPC) is calculated as follows:
TPC = (BT + BF + SC + MC)/(2*B + S + M) * 100%
where
BT - branches that evaluated to "true" at least once
BF - branches that evaluated to "false" at least once
SC - statements covered
MC - methods entered
B - total number of branches
S - total number of statements
M - total number of methods
What about Line Coverage metric?
A Line Coverage metric is a basic metric offered by bytecode instrumentation tools, such as Cobertura or Emma, and it's tightly related with a fact that inside compiled classes we can have only information about line numbers (instead of information about code elements, such as statements etc).
As Clover uses source code instrumentation, it actually "sees" a real code structure. Therefore, Clover offers a Statement Coverage metric, which is similar to a Line Coverage metric in terms of it's granularity and precision.
その他の情報