I work on a small/medium sized Java application. Over time, I've tried to ensure that the tests which I write are "good" tests, and that there's plenty of them. Hence I've been looking into various, more rigorous, testing methodologies, such as boundary value analysis, combinatorial tests, and other testing methods.
However, most of these have examples that are very dry, and the examples don't map very well to my domain, as the examples are mostly functions of numerical data. My functions are almost never concerned with numerical data, they're about enforcing business rules.
The application is hibernate/spring, and my testing strategy is as follows:
- Every single class gets a unit test. Collaborators of each class are entirely mocked away at this point, and as far as is reasonable, the contract of that class is exercised. Usually with hand-picked test cases. These tests often run in seconds.
- A full spring context is fired up by TestNG. All of a class' collaborators are therefore present, and even the database access layer is active (with a real, on-disk database that is trashed at the end of the test). Once again, the system is exercised using hand-selected test-cases. The start up time for these tests often stretches into 20 - 30 seconds apiece, and there's enough of them to seriously impact the build time.
- The system is booted up, usually against a blank database, and then a copy of an end-user database. A developer "clicks around" each booted system in an attempt to elicit a failure. This takes a while, easily over 30 minutes.
I worry that my testing regimen is insufficient, and that my tests are not as effective as their metrics would imply (it's quite easy to get close to 100% coverage with this scheme). There is no formal guidance on how to select test cases, aside from "what the developer thinks are good tests."
I've considered trying to express the behaviour of the system in something roughly like Z notation, and using that specification to drive test case production, but that is likely to be very costly in terms of time. Similarly, convincing my colleagues that there will be a reasonable return on investment of such an activity would be a very hard sell, and even I'm not convinced it's worth it for our system.
There have been discussions relating to randomised tests, but they've been vague and very general; and we rarely bump into run time errors/exceptions (NullPointerException, ArraysOutOfBounds, etc.), it's usually that the behaviour of the system is unacceptable to the client (i.e. in corner/degenerate cases, the system behaves in a counter intuitive manner, but which is completely consistent)
I would like to get the most "bang for my buck" with my tests; that is, how does one design test cases which are likely to exercise a defect in the systems similar to the one above?