Now I know people could consider this question duplicate or asked many times, in which case I would appreciate a link to relevant questions with answer to my question.
I have been recently in disagreement with some folks about code coverage. I have a group of people who want our team to drop looking at code coverage altogether based on the argument that 100% coverage does not mean good quality tests and thus good quality code.
I have been able to push back by selling the argument that Code Coverage tells me what has not been tested for sure and help us focus on those areas.
(The above has been discussed in a similar fashion in other SO questions like this one - https://stackoverflow.com/questions/695811/pitfalls-of-code-coverage)
The argument from these folks is - then team would react by quickly creating low quality tests and thus waste time while adding no significant quality.
While I understand their point of view, I am searching for a way to make a more robust case for code coverage by introducing more robust tools/frameworks that take care of more coverage criteria (Functional, Statement,Decision, Branch, Condition, State, LCSAJ, path, jump path, entry/exit, Loop, Parameter Value etc)
.
What I am looking for is suggestion for a combination of such code coverage tools and practices/processes to go with them which can help me counter such arguments while feeling comfortable about my recommendation.
I would also welcome any accompanying comments/suggestions based on your experience/knowledge on how to counter such an argument, because while subjective, code coverage has helped my team be more conscious of code quality and value of testing.
Edit: To reduce any confusion about my understanding of weakness of typical code coverage, I want to point out that I am not referring to Statement Coverage
(or lines of code executed) tools(there are plenty). In fact here is a good article on everything that is wrong with it: http://www.bullseye.com/statementCoverage.html
I was looking for more than just statement or line coverage, going more into multiple coverage criteria and levels.
See: http://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
The idea is that if a tool can tell us our coverage based on multiple criteria then that becomes a reasonable automated assessment of test quality. I by no means am trying to say that line coverage is a good assessment. In fact that's the premise of my question.
Edit:
Ok, maybe I projected it a bit too dramatically, but you get the point. The problem is about setting processes/policies in general across all teams in a homogeneous/consistent fashion. And the fear is general that how do you ensure quality of tests, how do you allocate guaranteed time without having any measure to it. Thus I like having a measurable feature that when backed up with appropriate processes and the right tools would allow us to improve code quality while knowing that time is not being force spent in wasteful processes.
EDIT: So far what I have from the answers:
- Code reviews should cover tests to ensure quality of tests
- Test First strategy helps avoid tests that are written after the fact to simply increase coverage %
- Exploring alternative tools that cover test criteria other than simply Statement/Line
- Analysis of covered code/number of bugs found would help appreciate the importance of coverage and make a better case
- Most importantly trust the Team's input to do the right thing and fight for their beliefs
- Blocks Covered/# of tests - Debatable but holds some value
Thanks for the awesome answers so far. I really appreciate them. This thread is better than hours of brainstorm with the powers that be.