6

Assume a method with the following declaration

public IList<Location> GetLocations() { ... }

I write (at least) the following tests

  1. Check to see if the method returns an empty collection if no locations are found.
  2. Check to see if the method returns the correct number of locations.
  3. Check to see if the data in the locations the method return is correct.

However, all the tests above will fail if the method returns null. I feel like a single error should only result in one unit test failure, but at the same time I do not see how to avoid it in a scenario like this. Is this just the way it works; sometimes multiple tests will fail for the same reason?

  • Are you sure you are testing the right thing? Seems to me that GetLocations() unit test should test whether it returns the data it got from (whatever data source you use), and not the actual data returned. – Wilbert Jul 22 '14 at 10:33
  • @Wilbert You what? This method takes no parameters, so it clearly just returns all the available data. Why would you not test that it returns all the objects/records, rather than just some, or an empty list if there are none? Tests 1 and 2 are absolutely valid. If test 3 means "check that the values for each location in the response corresponds to the locations in the source data", which I think it probably does, then test 3 is testing "whether it returns the data it got from (whatever data source you use)". Tests 1 and 2 do that as well, in fact. – itsbruce Jul 22 '14 at 10:59
16

Your assumption is wrong,

I feel like a single error should only result in one unit test failure

every unit-test test only one aspect of the method so:

  1. Check to see if the method returns an empty collection if no locations are found.
  2. Check to see if the method returns the correct number of locations.
  3. Check to see if the data in the locations the method return is correct.

in every above test if the method return null, test fail is the auspicable response

sometimes multiple tests will fail for the same reason?

The only acceptable answer should be Yes

| improve this answer | |
  • 4
    Absolutely correct. The point of a test suite is not that every conceivable defect should trip exactly one test, but at least one test. Then, as long as you only publish when the entire test suite succeeds, no defect can ever recur, and that's good enough. – Kilian Foth Jul 19 '14 at 17:21
2

I disagree with Infer-On's answer. Indeed, though not always achievable in practice, one should strive for having a single error cause just one test to fail.

The main fallacy of your assumptions is in your list of behaviors. There is one crucial test missing from your list based on the information you provided:

0. Check to see if the method returns null when circumstance x occurs.

This is the one and only test where your method should be allowed to return null. If it returns null in the other 3 tests you have listed, then those tests are not testing what they claim.

First, you need to make sure your behaviors (tests) are correctly stating what they want. Roy Osherove in The Art of Unit Testing goes into this in copious detail but to be very brief, he purports a test name should be of the form <method> <scenario> <behavior>. And, in fact, your first test abides by this, just in a slightly different order, which is perfectly fine (and the way I tend to do it, as it happens). Here I have separated out the components in your test name:

1. <method> <returns an empty collection> <if no locations are found>

But your second and third tests are missing the scenario.

2. <method> <returns the correct number of locations> <when what??>
3. <method> <returns correct location data> <when what??>

Let's say that the scenario (the "when what??") for both (2) and (3) is something like "when given proper coordinates". (Replace that with whatever fits in your context.)

Getting back to the main point, though, for all 3 tests, if the method is returning null then it is not given the opportunity to look for locations and find none (1) and it is not given proper coordinates (2) and (3). So those tests are failing but for the wrong reasons.

So to put into practice: The extra test I introduced above (0) handles the case of the method returning null, so the method must not return null in the remaining tests. You must instrument them appropriately with test doubles (typically mocks and/or stubs) so you can eliminate outside influences and isolate precisely what you want to test.

| improve this answer | |
  • 1
    I'm afraid I don't understand. You make it sound as though the remaining three tests (the original ones) should only be run based on the result of the test you added, which goes against each test running in isolation. I can test to see if the method returns null, but that doesn't change the fact that the other three tests will fail. And since the other three tests will fail the null check method is redudant as well. – user1323245 Jul 22 '14 at 4:34
  • 2
    @user1323245 Null is clearly completely invalid output given your description;the method should always return a list, either empty or populated. If there's some error in its input (or its interaction with other methods), the method should be throwing an exception. How could Null be useful output? So a test for Null absolutely should be there. And what is the point of testing the correctness of the output if the output is simply completely wrong? There's no law that says all tests should be performable in parallel. Some tests, if they fail, clearly make other tests pointless. – itsbruce Jul 22 '14 at 8:09
  • If you do feel that Null is acceptable output in some circumstances (which seems odd to me in this context) then you should be testing that it does return Null in those circumstances. That should be one of the minimal set of tests, but you don't mention it in your minimal list - "I write (at least) the following". – itsbruce Jul 22 '14 at 8:15
  • I believe my thinking has been off. I was somehow caught on the idea that if the method return null it will be obvious in the existing tests. However, that may or may not be true, and it may change in the future, but most importantly, it's something that is only known by actually looking into those tests methods. Clearly a null check method should exist. Thanks – user1323245 Jul 22 '14 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.