Sometimes we have code like this:

private static final List<Goat> INEDIBLE_GOATS = [GOAT_A, GOAT_B, GOAT_C] 

void eatGoat(Goat goat) {
   if(INEDIBLE_GOATS.contains(goat)) {
       throw new CantEatGoatException();

Should it best tested (along with whatever tests for the actual eating) as:

for(Goat goat: INEDIBLE_GOATS) {
   // forgive abuse of syntax
   assertThrows(CantEatGoatException, eatGoat(goat));

Or more like?

assertThrows(CantEatGoatException, eatGoat(GOAT_A));

Should tests verify each and every sort of inedible goat? Or should they test a representative goat from the list and trust that it works for the other goats?

  • 3
    ROFL. Why do I get visions of farmers' sons doing unthinkable things with the livestock when nobody's looking? May 12 '14 at 23:42
  • In other news, what coding convention is this? Makes my eyes burn. May 12 '14 at 23:43
  • Shouldn't your test include both the positive and negative results? assertTrue(isFoobarableGoat(new Goat()) and assertTrue(isFoobarableGoat(new FoobarableGoat()) May 12 '14 at 23:55
  • @AdamZuckerman, obviously. But I didn't include them to reduce TLDR factor. May 12 '14 at 23:59

A major reason to have tests is to build your confidence in your code. Tested code can be proved to have a specific functionality because the tests say they do. Confidence grows from the proof. The question is, by looping through all the INEDIBLE_GOATS, do you gain any more confidence in your code? Either way, with both positive and negative tests, you cover the relevant code. With the loop, though, you are basically exercising these lines over and over. Essentially, you are testing that List<Goat>#contains() does what it says. Do you gain anything from that? If you trust that a List works as it should, then no, you don't get anything additional. If, on the other hand, you are not confident that a List works as it should, then you should have a separate test of List#contains() to give you that confidence.

You also have to balance the number of tests with the performance of your test suite. Looping through all the INEDIBLE_GOATS may give you a little more confidence, but if there are, say, 1.8 million of them, and running the tests takes 30 seconds longer, does that tiny additional confidence outweigh the lost time (and ensuing lost productivity).?

Finally, while INEDIBLE_GOATS may only have a small number of elements now, theoretically, the number of inedible Goats may actually be infinite. It is impossible to test every potential Goat. While in this simplified example, you may say that there won't really be infinite Goats, in general, we must account for infinite and near-infinite possible inputs and accept that we cannot practically test everything. We must also realize that this is a good thing. Not all inputs are equal, and trying to test everything wouldn't necessarily get us any more confidence. It is optimizing tests for confidence that makes them practical and profitable.

  • 2
    As an aside, I tend to test code that uses complex regexes pretty thoroughly as they are fragile and hard to read. The cost of thorough testing is worthwhile because small, hard-to-notice-or-decipher changes can break a regex, and knowing that as soon as possible helps to speed fixing the bug.
    – cbojar
    May 13 '14 at 4:41

The second test seems more accurate to me. If you put a Goat within your List, you expect that Goat to be a foobarable Goat (that is, you expect to get true from your method).

  • What does "more accurate" mean? May 12 '14 at 23:46
  • I assume the test he's talking about deals with the method boolean isFoobarableGoat(Goat goat). So, if he calls that method for a bunch of goats already contained within the list, he cannot be sure the method works the other way as well. Maybe he should add to his test something like assertFalse(isFoobarableGoat(outerGoat));.
    – b3by
    May 12 '14 at 23:50

Seems like you're just pushing the logic around. How about:

  • 1
    The whole point is to test isFoobarableGoat. So its kinda not helpful to not call that function. May 12 '14 at 23:49
  • 2
    Yeah, this is what happens when your examples are too abstract. May 12 '14 at 23:50
  • I've tried to deabstract it a bit. May 12 '14 at 23:58
  • Does the collection only have to contain one FoobarableGoat, more than one, or does the entire collection need to contain FoobarableGoats? May 13 '14 at 0:00
  • There is only one collection and its static final. Its part of the code and always has sone exact number of arguments. May 13 '14 at 0:03

Your question is an important one. To restate more generically: should you test all possible inputs or just selected ones? As @cbojar points out, the former is typically either (a) not practical or (b) not useful.

The question then becomes how to whittle the list of all possible test inputs into a set of optimal test inputs. To do this, divide the set of all possible inputs into groups based on expected outcomes. Any input from one group should yield equivalent (though not identical) results. Such groups are called equivalence classes. and this process of identifying such classes is equivalence partitioning.

As a simple example, to test a square root function that takes an integer input, a good starting point would be these equivalence classes:

 1   -- all positive integers
-1   -- all negative integers
 0   -- zero

You might refine that list further:

 1   -- all positive integers that are not perfect squares
 4   -- all positive integers that are perfect squares
-1   -- all negative integers
 0   -- zero

And if you are a real stickler, you might add maxint and minint for your integer type. But with any of these you have infinitely less test cases than the set of all possible inputs :-) -- a pretty good savings!

(For more, see my answer to a related question, What should you test with unit tests?)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.