3

I work on a small/medium sized Java application. Over time, I've tried to ensure that the tests which I write are "good" tests, and that there's plenty of them. Hence I've been looking into various, more rigorous, testing methodologies, such as boundary value analysis, combinatorial tests, and other testing methods.

However, most of these have examples that are very dry, and the examples don't map very well to my domain, as the examples are mostly functions of numerical data. My functions are almost never concerned with numerical data, they're about enforcing business rules.

The application is hibernate/spring, and my testing strategy is as follows:

  1. Every single class gets a unit test. Collaborators of each class are entirely mocked away at this point, and as far as is reasonable, the contract of that class is exercised. Usually with hand-picked test cases. These tests often run in seconds.
  2. A full spring context is fired up by TestNG. All of a class' collaborators are therefore present, and even the database access layer is active (with a real, on-disk database that is trashed at the end of the test). Once again, the system is exercised using hand-selected test-cases. The start up time for these tests often stretches into 20 - 30 seconds apiece, and there's enough of them to seriously impact the build time.
  3. The system is booted up, usually against a blank database, and then a copy of an end-user database. A developer "clicks around" each booted system in an attempt to elicit a failure. This takes a while, easily over 30 minutes.

I worry that my testing regimen is insufficient, and that my tests are not as effective as their metrics would imply (it's quite easy to get close to 100% coverage with this scheme). There is no formal guidance on how to select test cases, aside from "what the developer thinks are good tests."

I've considered trying to express the behaviour of the system in something roughly like Z notation, and using that specification to drive test case production, but that is likely to be very costly in terms of time. Similarly, convincing my colleagues that there will be a reasonable return on investment of such an activity would be a very hard sell, and even I'm not convinced it's worth it for our system.

There have been discussions relating to randomised tests, but they've been vague and very general; and we rarely bump into run time errors/exceptions (NullPointerException, ArraysOutOfBounds, etc.), it's usually that the behaviour of the system is unacceptable to the client (i.e. in corner/degenerate cases, the system behaves in a counter intuitive manner, but which is completely consistent)

I would like to get the most "bang for my buck" with my tests; that is, how does one design test cases which are likely to exercise a defect in the systems similar to the one above?

  • 4
    What actual, real-world difficulties are you having with your existing code coverage? Are the tests missing things? What sort of things? Because it sounds to me like you're already above typical standards, unless you're writing software for a space probe. – Robert Harvey Jan 17 '15 at 19:41
  • While we do face some run-time exceptions, the usual problems which we have revolve around the end-user seeing behaviours which are simply unexpected. We also face issues which are configuration-related. One part of the system grants access rights (defined in configuration) which more limited than the "reader" of the data was expecting. An end user sees this as a security exception. – Tinned_Tuna Jan 17 '15 at 19:57
  • We're not writing high assurance code, like for space craft, but we do drive business processes for our clients; process which, if it is not followed, could lead to the closure of their business. – Tinned_Tuna Jan 17 '15 at 20:03
  • 2
    For example, we have a "waiting list" feature, users may wait for a place to become at an event (events may run multiple times, and users tell us what times they are available). When one does, they are offered that place. Unfortunately, this means that if a user gives us times which never become available, we end up with a user permanently on the waiting list, and the list can become very long. To many people this is counter-intuitive, that a person can wait indefinitely, but it seems reasonable that we give offers when the're available, and don't surprise users by kicking them from the list. – Tinned_Tuna Jan 17 '15 at 20:13
  • 6
    Some things you can't forsee. Having waiting list items that never time out may make sense to you, but may not make sense to the customer. That's why iterative development processes exist. Fundamentally, I don't see your waiting list problem as a testing issue. – Robert Harvey Jan 17 '15 at 20:17
1

Further to class level unit tests, what's in my opinion perhaps a more grounded test and pragmatic approach is to execute all the critical/high value execution paths..

identify and create a metrics (of criticality) for your end user's minimum required critical functionality set - invoke these functions/operations at (increasingly) much higher level than at a class level. A state transition comparison against an expected state would be your test. For example state transition could be change in database table, addition in a message queue and so on - obviously that is the expected end result of the functionality under testing.

Unit tests are good, but they are also expensive. Writing and CERTAINLY maintaining one in a fast evolving product environment has very serious and significant cost. Developers can also go to extremes to test at unit level and in the process end up loosing the far sight.

100% coverage is a very subjective term as you already know.

0

I have always found if you put tests into the running code, even in production (ones that do not degrade performance), gives you more results for your testing money, than the heavily isolated unit tests that are currently in fashion. The tests I speak of are based on the design of each method and class. In each important, externally accessed method:

  • Test preconditions.
  • Test postconditions.
  • Test class invariants.

If you don't know what the preconditions, postconditions, or invariants should be, then the problem is in the design phase. In this case, I recommend to read more about these concepts, and why they are important.

While this breaks the ideal that some strive for (isolated tests), any testing of a higher level component will test lower level components as well, so you get more testing accomplished overall. Specifying preconditions, etc, is also good for designs in general. It is never good to throw code to customers without understanding its limitations.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.