Should we exclude code for the code coverage analysis?

Question

I'm working on several applications, mainly legacy ones. Currently, their code coverage is quite low: generally between 10 and 50%.

Since several weeks, we have recurrent discussions with the Bangalore teams (main part of the development is made offshore in India) regarding the exclusions of packages or classes for Cobertura (our code coverage tool, even if we are currently migrating to JaCoCo).

Their point of view is the following: as they will not write any unit tests on some layers of the application (1), these layers should be simply excluded from the code coverage measure. In others words, they want to limit the code coverage measure to the code that is tested or should be tested.

Also, when they work on unit test for a complex class, the benefits - purely in term of code coverage - will be unnoticed due in a large application. Reducing the scope of the code coverage will make this kind of effort more visible...

The interest of this approach is that we will have a code coverage measure that indicates the current status of the part of the application we consider as testable.

However, my point of view is that we are somehow faking the figures. This solution is an easy way to reach higher level of code coverage without any effort. Another point that bothers me is the following: if we show a coverage increase from one week to another, how can we tell if this good news is due to the good work of the developers, or simply due to new exclusions?

In addition, we will not be able to know exactly what is considered in the code coverage measure. For example, if I have a 10,000 lines of code application with 40% of code coverage, I can deduct that 40% of my code base is tested (2). But what happen if we set exclusions? If the code coverage is now 60%, what can I deduct exactly? That 60% of my "important" code base is tested? How can I

As far as I am concerned, I prefer to keep the "real" code coverage value, even if we can't be cheerful about it. In addition, thanks to Sonar, we can easily navigate in our code base and know, for any module / package / class, its own code coverage. But of course, the global code coverage will still be low.

What is your opinion on that subject? How do you do on your projects?

Thanks.

(1) These layers are generally related to the UI / Java beans, etc.

(2) I know that's not true. In fact, it only means that 40% of my code base

Answer 1

I generally exclude auto-generated code, such as the WCF clients that Visual Studio generates. There are usually a lot of lines of code there and we're never going to test them. This makes it very demoralising to increase testing on a large chunk of code elsewhere and only increase code coverage by 0.1%.

I will also exclude data-layer interactions, as long as the team can say with certainty that this layer is as thin as it possibly can be. While you can argue that, if the layer is thin, it won't have a massive effect, it does leave a lot of components in the coverage report with 0% against them, so we don't necessarily notice the ones that we need to really worry about. The UI layer could be argued in a similar way, depending on the framework being used.

But, as a counter-point, I will also exclude the unit tests themselves. They should always have ~100% coverage and they amount to a large percentage of the code-base, skewing figures dangerously upwards.

Answer 2

Good question. In general I tend to exclude code from code-coverage for some reasons, e.g. it is:

• generated
• legacy (not updated anymore)
• here it comes: certain layers, I don't intend to test

Why the last point? I think it's good practice to focus on things I care about, while suppress distractions. If you don't intend to test the UI-Layer, why take it into consideration - it would only draw attention away from the important part of your software - the business logic.

BUT:

1. You should have a good reason, why excluding a certain layer from unit-testing at all (there may be questions - from your boss, your teammates, the press ;-)
2. If you want them to test these layers, you should include them in the statistics to show the whole team, every day that it is important and needs to be done.

Finally: don't take numbers too serious. 30% coverage may proof a software rock solid, when it is the essential part of it.

Answer 3

I tend to agree with you, and include all relevant code in code coverage metrics (and Sonar in general). Even if you don't plan to test some parts of the code (for the foreseeable future that is), the metrics should reflect the actual status. This forces you to have a compelling reason not to write tests for a significant part of the codebase. Eventually (once other, more important parts of the code are already covered) you may reconsider, or choose a different way to eliminate this dissonance - e.g. by refactoring the code in question to make it testable, or to migrate all logic from it into a different, testable part of the codebase, or even to eliminate the whole layer altogether (if a layer is not worth testing, does it have a good enough reason to exist?)

In my current projects, we also include all the code in metrics, with one exception: generated code, producing heaps of static analysis warnings. As this code typically consists of humongous POD classes without any logic, there is nothing to test there anyway.

Answer 4

Now I am not much acquainted with the code coverage tools that you are using nor am I familiar with the Java beans, but from what you say they are related to the UI.

With my limited knowledge I have this to say:

1. Do not let numbers by any kind of testing tools hinder with your tests.
2. If the classes are complex and not unit testable, it is always a good idea to re-factor them into more compact and testable classes. It will require a lot of effort and dedication but it will pay in the long run.
3. Testing UI components can be tough, but you could still test the function that is being performed by those components.
4. If you are doing a web based project then you can use QUnit to unit test all of the client side code.

All in all you should keep in mind that code coverage is just a number, and high code coverage does not indicate good tests. Focus on making the core libraries more robust and testable rather than aiming for a higher percentage.

Answer 5

Some exclusions make sense (boiler plate code that has no real impact or potential of impact on your project functioning correctly). Even still collecting code coverage as a metric is pointless because it doesn't tell you anything useful anyway. 100% code coverage isn't a real goal, and low coverage numbers also may not be bad depending on the project, it could be possible that 20-30% code coverage covers everything that is worth testing. code coverage only tells you X% of your code that is potentially worth covering actually is by some type of test of unknown quality.

Answer 6

I suggest reporting a set of metrics for each layer of your code. These metrics should include size information (e.g., LoC, number of libraries, number of classes or methods, etc.), testing information (e.g., coverage), and bug information (e.g., find and fix rates).

Your excluded layers will have 0% coverage, and your tested areas will have the 60% coverage you mentioned. The size information will let people understand how much is untested or tested. The bug information will inform you if you maybe should be creating tests for the excluded layers, and if your existing tests are working.

It is easy for software organizations to become too focused on the purity of metrics. We don't make metrics, we make good software. A metric that that helps you to deliver high-quality software in a timely manner is the most honest, pure metric there is.