Code Exploration (CE) is quite a new term and I wonder if there already any successful examples of implementing this techniques in terms of Continuous Integration principles?
In short, Code Exploration can be described as the process of testing your code using different parameters to achieve high code coverage. The one-and-only tool for .NET world is PEX, but I know there are commercial instruments for code exploration in java world.
This topic is very powerful in conjunction with Design By Contract, because it provides you with the means of some formalism in reasoning about code correctness. For example PEX would show that next code contains error (source code uses Code Contracts and I assume default assembly configuration):
public int Factorial(int number)
//postcondition of positive result, precondition is omitted
Contract.Ensures( Contract.Result<int>() > 0 );
return number <= 0 ? 1 : number * Factorial( number - 1 );
Because by default overflow is not checked in arithmetic operations the input 17 will give the result of -288522240.
PEX was discussed here several times here:
But I don't want to touch internal implementation details, I'm interested in practical aspects of this problem.
Main questions are:
How exactly do you integrate Code Exploration with Continuous Integration?
Do you reject build if PEX gives errors or you notify team about inconsistencies and contracts violations?
Scott Hanselman has point, that PEX should run on separate machine (or multiple machines) continuously testing your code for errors (so called Continuous Exploration). You can listen to the talk Hanselminutes Podcast 93 - Pex. Are there any practical implementations of dedicated machine (would it be virtual or physical), that digs into your codebase in search of errors?
How do you treat test coverage of such tools? I think it would be unreasonable to merge code coverage results for unit tests written manually with the results by a code exploration tool.
Are there any examples of Parameterized Unit Tests in real projects, that would increase you productivity and unit tests quality?