6

Our project uses Jacoco for code coverage metrics. Previously, we have checked the line and branch coverage against percentage values. However, the documentation also describes an "Instructions" level, which counts Java byte code instructions. From their documentation:

The smallest unit JaCoCo counts are single Java byte code instructions. Instruction coverage provides information about the amount of code that has been executed or missed. This metric is completely independent from source formatting and always available, even in absence of debug information in the class files.

If we have been using line and branch coverage previously, what is the advantage of switching to instruction coverage? Is it just counting the same things in a different way? Or does it provide more accurate metrics? Conversely, is there a situation where we would want to use line and branch coverage metrics instead of byte code?

Improve this question
0

3 Answers 3

Reset to default
7

There is no advantage of byte-code or line coverage over branch coverage.

If one element (line or instruction) of a branch is covered, then all elements of that branch are covered as well.

If a single line of code contains more than one branch (e.g., in C: (a==0) ? do_b() : do_c()), branch coverage takes it into account.

Moreover, branch coverage is completely independent from source code formatting.

Improve this answer
5

The advantage the docs mention is that line coverage counts

x = 1; y = 2; z = 4;

as one line, but 

x = 1;
y = 2;
z = 4;

as three. On the plus side, bytecode-based measures would not vary between the two, because the two versions compile to the same bytecode sequence. Line-based metrics would report different numbers.

Bytecode coverage as a reporting metric seems gimmicky. Having a metric independent of code formatting is certainly a virtue in theory. It might be a plausible defense against developers gaming coverage numbers, say by putting many logical statements on one line. But it depends on language implementation details, not the source code. It wouldn't provide stable numbers across different implementations of Java, some of which don't even have bytecodes, and would even vary over time as the JVM and compilers evolve. Most if not all testing engines already discount the most obvious formatting deltas, such as comment and blank lines. Code reviews and coding style standards also already naturally discourage gratuitous line-packing and other coverage metric-gaming.

Bytecode coverage is not a standard or widely-accepted metric / unit of reporting. Unlike branch coverage, counting bytecodes is not a "subtle, but meaningful" semantic distinction in the coverage being measured. It is just an alternate, less-formatting-sensitive way to measure / report the same thing line coverage already measures. It has some slight advantages, but they don't seem especially large when balanced against them being a nonstandard measure, tied to a specific (if popular) language implementation, and not directly correlated to visible program source.

Improve this answer
7
  • 1
    Right. For example, Oracle's javac is pretty simple and doesn't perform many optimizations. But a more aggressively optimizing compiler might notice that, say, y isn't used at all, and x is only used much later, and thus delay the initialization of x and remove y altogether. What does instruction coverage tell you then?
    – Jörg W Mittag
    Jul 17, 2017 at 23:14
  • 1
    I can see this being useful in cases where JVM bytecode is the programming language, e.g. for bytecode written by hand, or libraries that generate bytecode (e.g. Kilim), where you want to ensure that functional tests fully cover the generated bytecode.
    – Jörg W Mittag
    Jul 17, 2017 at 23:16
  • If you have debugging information, couldn't the tool generate "expression level" coverage metrics using "instruction level" coverage? That would be actually useful, although going a bit deeper into the VM guts.
    – Mael
    Jul 18, 2017 at 9:40
  • How would expression coverage be useful? Also, if you start doing expressions, won't you next want subexpressions? Hello, combinatorial explosion!
    – Jonathan Eunice
    Jul 18, 2017 at 15:21
  • @JonathanEunice It was just an idea that came into my head after I saw "instruction coverage", and apparently I didn't think it through. I was thinking about (X ? Y : Z) it is one line, but three expressions (although it's really one expression consisting of three subexpressions, as you pointed out).
    – Mael
    Jul 18, 2017 at 22:21
2

Java byte code interpreters are considered an implementation detail. You shouldn't have to test or assess metrics at the byte code level, since you'd essentially be testing for bugs in the byte code interpreter (which you really shouldn't have to worry about). I read the paragraph you linked as "if all you have is the byte code binary, you can still get metrics this way."

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.