test coverage reality

Question

I am NOT doing test driven development and I write my test classes after the actual code is written. In my current project I have a test coverage (Line coverage) of 70% for 3000 lines of Java code. I am using JUnit, Mockito and Sonar for testing. But I feel that I am not actually covering and catching 70% of the problems that can occur.

So my question is in theory is it possible to have a 100% Line coverage, but in reality it is meaningless because of low quality of the test code and maybe a 40% well written test code is much better than a bad 100% coverage? Or we can always say line coverage more or less gives the percentage of all covered issues?

Answer 1

There are different ways to count coverage. Many tools check which lines are executed by tests, but they don't check that the tests actually test anything. So in theory you could have 100% coverage without a single assert in your tests, in effect testing nothing.

However, you say you have 70% coverage, but you don't catch 70% of problems. That is not uncommon, since quite often small subset of your code contains most of the problems. I think Code Complete says something along the lines that 20% of the code causes 80% of the bugs. That problematic area in your code is usually complex and that is why it's usually untested. So you might have good code coverage but the part that is not covered is the part that needs the tests most.

In this article Martin Fowler says test coverage is not a good measurement for quality of your code. It is a good way to find problematic areas in your code. So in your case, assuming your existing tests are good tests, you now should be able to pinpoint the problem area in your code - the remaining 30%.

Answer 2

You're right, a line coverage metric is no guarantee that you have a comprehensive test. Path coverage (ensuring that every possible route through the code is exercised at least once) is far more important. However, path coverage is far more difficult to calculate than code coverage is, and code coverage is nonetheless useful (while a line that is covered isn't necessarily comprehensively tested, a line that isn't covered definitely isn't tested at all).

You should pay close attention to the points in your code where the flow of execution branches. Ternary operations (var = foo()? bar(): baz()) should be of special interest because these are the points that can appear fully covered even when only one of the two available execution paths are followed.

It should also be mentioned that when a portion of the code is not covered by tests, it's often because it's a difficult portion of code to test for various reasons (extremely complicated logic, dependencies that can't be easily mocked, dependant on probabilistic events, intended to generate a probabilistic output, etc). This is often also the kind of code where testing is most important. If a module has 95% code coverage in its tests that's still not much use if the uncovered 5% is the part of the code that's mostly likely to harbour bugs.

If code is well designed with testing in mind (cohesive, loosely coupled, utilises dependency injection, etc) then very high code coverage of 90% or higher is definitely possible. But of course, while it's a useful metric, a high coverage figure doesn't mean your code is well tested.

Answer 3

I don't think there's a general correlation between coverage and bugs left. However, if you didn't execute a line, you didn't test it, so 100% line coverage can't hurt if you don't let it lull you into a false sense of security.

In the end, unless you're writing a system with extremely high reliability requirements, I wouldn't sweat too much about raw coverage. First, test the most important paths through the program. Then, for every issue that crops up, write a unit test before fixing it. That way, you will get decent coverage in a reasonable time and will be quite safe from regressions.