Unit testing coverage

Question

How much of your code do you unit test? Do you try to get 100% coverage of everything or do you skip some classes?

Background

We've made a lightweight ORM layer which we have written plenty of tests for. On top of that we use Code Contracts to specify what each method in every interface should accept and return.

Classes to test or not to test

We create repository classes to fetch items or collections. These classes are basically SQL queries and calls to the ORM layer. We also have entity mapping classes similar to those in fluent nhibernate.

Why I ask

The reason I'm asking is that I don't really see the point in writing unit tests for these classes. Well, there is a point if we could use a test database to see that no exceptions are thrown. Our test databases are a bit messy and are never reseted, hence it's impossible to validate any data (and that is out of my control). We could use a fake/mock ORM layer to try the repositories, but I do not see the point in it.

Answer 1

I agree with Peter. It sounds like your getting the gut feeling as to something you don't think is worth testing but you know from a text book, that the more testing the better.

Firstly, 100% test coverage is impossible and not worth the effort in trying to achieve. 85% or over, and you're doing well.

As for your ORM framework, again, it's up to you. Will you allow developers to extend/change behaviour of anything generated, if so, it might be viable to wrap those up in some generated tests. If it's a closed book, then what's the point?

Generated tests to hit the proc's in the database, where they could be modified by developers/DBA's (and usually are), again, it might be nice to have a generated test for that if it's not too difficult to do.

It's always a trade off - is it worth the effort? This also usually comes up when people consider writing tests over entity classes with just get/setters on them...

Answer 2

In writing unit tests, you should always use your own judgement on what and how much to test. No textbook rule can fit every specific project and situation. We have limited resources, which we should use sparingly to get the most benefit for the given time and effort. Ther are always code parts which can't be reasonably unit tested, but there may be better ways to test them, such as integration / system / functional tests.

Even on the classes you do unit test, it is hardly reasonable to expect 100% coverage. There is always error/exception handling code etc. which is hard to test and doing so doesn't bring much benefit (if at all).

So for every class / method you (or prefereably the whole dev team) should ask yourself whether you are confident enough that this code works. In other words, can it be shipped today? If not, work out a way to ensure its quality, be it unit or other kinds of tests.

Answer 3

Test coverage doesn’t really say anything. It says that you test the class, but it doesn’t say anything about all permutation (different scenarios) that you can test the class.

EDIT

For example you have x fields with y different state each in an object. These states generate different business scenarios (rules). IMO a meaning full unit test on the class is one that covers all possible permutation of these states. A more complex scenario is when this object interacts with other object. In such case the permutation states for your class must also be combined with other classes. A realistic scenario is to narrow down to all possible scenarios, in such case you take only those permutations that are possible in you test.

Answer 4

Everyone has their own comfort level (although some are a bit too comfortable), but for me, I strive to get as close to 100% business logic coverage as I can. I write mostly financial software so this could be for things like checking that line items add up to a total, etc.

Also, keep in mind that 100% test coverage is unnecessary, since we honestly do not strive for 100% error free software. Before people grab the pitchforks, what I mean is that we strive to stop bugs within an acceptable scope. Someone could try to run my internal financial app through their home office Virtual Machine which is on a Solaris OS. Well it just so happens that we run on the Microsoft stack, so despite the fact that my app "should" run in the Solaris VM, I could care less that it doesn't, even if its totally my apps fault. That case would be outside the scope of unnacceptable errors, so I won't test for it. What needs to be tested is even a subset of possibilities within the list of unnacceptable errors. More than simply being unnacceptable, they must be somewhat likely to occur. No need to test that the compiler will break and misread your code or anything like that.

Answer 5

Code that ships, should be tested, or you will ship bugs. However, Unit Tests are just one tool to test code.

I've worked on projects that regularly shipped bug free with unit test code coverage around 20%, which also made use of manual test scripts and intelligent testers. In contrast I am currently working on a project that has a code coverage around 85%, which relies much heavier on unit tests to achieve a similar result.