Does code coverage make sense when running end-to-end tests?

Question

I have a situation where I am running mostly end-to-end tests on my framework:

• I start of with simple fixtures (which are CSV-files).
• These are imported by my application (which I consider to be a black box in my situation).
• When the importing is complete I check with repository classes if they return what I expect to be returned, and in some cases I direct query the database to see if things are as expected.

Now I think code coverage is great, but in this situation (PHP/PHPUnit/XDebug) it makes the executing of my code 50-80% slower (because XDebug is in-between for the code coverage aspect).

So what makes more sense in this case?

• Disable code coverage and have a quicker testsuite that actually tests with real-world scenarios, but without knowing if everything is covered codewise?
• Have a slower test but more insight in what's being tested (and what not?)

To put things in perspective: I now have only somewhat like 6 end-to-end tests which can take up to 12 minutes when code coverage is on, and about 2-3 minutes without code coverage.

What are your thoughts on this?

Answer 1

When testing you should not focus on code coverage but on requirement coverage.

Code coverage tells you if there is a test case employing a particular line of code. It does not tell you if that line of code actually contributes to the tested feature or not. That means you can have "covered code" which is essentially useless.

On the other hand especially when doing end-to-end tests you cannot test everything because you simply run out of time. Therefor in end-to-end tests code coverage makes no sense at all.

And finally you don't sell code to your customer, you sell functionality.

After all code coverage gives you a feeling for how trustworthy your unit-tests are. But it should not have high importance.

Answer 2

Can you readily turn coverage testing on and off? If so, you could do occasional coverage tests, but do most of the functional testing with it off.

Code coverage tests are good if you can do them, as they will reveal parts of the functionality of the code that you are not testing. If you're not testing it, you don't know whether that code works or not.

Answer 3

Code coverage is typically only useful when getting feedback on Unit Tests. Unit tests are where you want to test every edge case and handle the minutia of keeping each unit correct.

With end-to-end tests, you aren't typically exercising every final minute detail of your application. It's just not that informative. I'm inclined to recommend turning it off on end-to-end tests, assuming you have unit tests that you do have it on for.