15

To give a slightly contrived example, let's say I want to test that a function returns two numbers, and that the first one is smaller than the second one:

def test_length():
    result = my_function()
    assert len(result) == 2

def test_order()
    a, b = my_function()
    assert a < b

Here, if test_length fails, then test_order will fail too. Is it a best practice to write test_length, or to skip it?

EDIT: note that in this situation, both tests are mostly independent from each other, each one can be run in isolation, or they could be run in reverse order, this does not matter. So none of these former questions

is a duplicate of the one above.

Improve this question
11
  • 2
    possible duplicate of Unit testing - one test is partly a "superset" of another, is this wrong?
    – user53019
    Jan 12, 2015 at 15:41
  • 2
    @GlenH7 - this seems like a different question. The other is "if you have functions where A calls B and returns the same result, should you test both A and B". This is more about the tests being overlapping rather than the function(s) under test. (Though it's confusing as they're currently named).
    – Telastyn
    Jan 12, 2015 at 15:46
  • possible duplicate of How should I test the functionality of a function that uses other functions in it?
    – gnat
    Jan 12, 2015 at 17:03
  • 1
    Stricly speaking these tests aren't really overlapping (they don't have success dependency). A function lambda: type('', (), {'__len__': lambda self: 2})() will pass the first, but not the second.
    – liori
    Jan 12, 2015 at 19:40
  • 1
    @GlenH7: FWIW, I did not really change the question, just tried to make it gnat-safe ;-) (@gnat: no offense, just joking!)
    – Doc Brown
    Jan 13, 2015 at 16:02

4 Answers 4

Reset to default
27

There can be value, but this is a bit of a smell. Either your tests aren't well isolated (since test_order really tests two things) or you're being too dogmatic in your testing (making two tests testing the same logical thing).

In the example, I would merge the two tests together. Yes, it means you have multiple asserts. Too bad. You're still testing a single thing - the result of the function. Sometimes in the real world that means doing two checks.

Improve this answer
6
  • I upvoted your answer though it misses one minor point. In Python, the second test will also fail when my_function does not return exactly two values, without any assert - because the assignment will result in an exception. So there is actually no need to use multiple asserts and two checks to test the same thing.
    – Doc Brown
    Jan 12, 2015 at 15:58
  • @DocBrown - I assumed as much, but I'm not familiar with Python's error messages to know if there may be value in having an assert failure instead of random exception/error for informative reasons.
    – Telastyn
    Jan 12, 2015 at 16:00
  • I think in the current situation the error message would probably be informative enough. But in general, that might not be the case. That's why I agree to you that the general approach for a case like this should be to think of a "merge", using two asserts. If it turns out that the test can be further simplified by leaving out one of the assert, fine.
    – Doc Brown
    Jan 12, 2015 at 16:06
  • 5
    @DocBrown The other value in explicitly checking with an assert is that it becomes clear to anyone examining the test that this is a failure of the code under test, and not the test itself. When my tests encounter unexpected exceptions, I always have to spend some time figuring out which one is actually wrong.
    – Chris Hayes
    Jan 12, 2015 at 21:49
  • @ChrisHayes: I wrote "there is no need", not "there is no value" in it.
    – Doc Brown
    Jan 12, 2015 at 22:09
5

Your tests should be explicit. It's not completely inferred that if text_length fails test_order fails.

I'm not sure how it goes in Python which you've posted, but if len(result) is 3 then the first will fail but the second may pass (and if not in Python, then in languages like JavaScript for sure).

Like you said, you want to test that the function return two numbers and that they are in order. Two tests it is.

Improve this answer
7
  • 1
    It's not completely inferred that if text_length fails test_order fails - in Python, it is, will give you an exception.
    – Doc Brown
    Jan 12, 2015 at 15:59
  • I think the question is about the case where failure of test_length implies failure of test_order. The fact that a similar pair of tests written in Javascript wouldn't behave the same as these two python tests is kind of irrelevant.
    – Dawood ibn Kareem
    Jan 12, 2015 at 19:36
  • 2
    @DavidWallace: remember, the question was "Is it a best practice to write test_length, or to skip it"? In Javascript, you need both tests to make sure you did not miss an issue (when you don't merge the two tests into one). In Python, you can safely omit the first (which is denied in the first sentence of this answer, and declared as "I am not sure" in the second). Honestly, a good answer looks different. However, I did not downvote this answer since the good part actually is the mentioniong of JavaScript.
    – Doc Brown
    Jan 12, 2015 at 20:29
  • 4
    @DavidWallace: since we are here an programmers.com, not on stackoverflow.com, IMHO giving an answer which could be applied to more than one language is better than an answer only for a particular language. But when referring to a specific language, the answer should be correct about the language, and not mix up some properties.
    – Doc Brown
    Jan 12, 2015 at 22:25
  • 1
    Thing is, the question is "is it worth writing a test that's a subset of another test", and this answer says, "in some languages, neither of your tests is a subset of the other" and then reaches the conclusion both tests are therefore necessary. It reads to me as if someone said, "your code doesn't compile at all in C++, and besides, in C++ a function can only return one value so you don't need to test it returns two. Only write one test" ;-)
    – Steve Jessop
    Jan 13, 2015 at 9:44
2

The only value of test_length here is that, if everything passes, its very presence indicates that "length" has been tested.

So it's really unnecessary to have both these tests. Keep only test_order but consider renaming it test_length_and_order.

Incidentally, I find the use of names that begin with test a bit clumsy. I'm a strong advocate of test names that actually describe the condition that you're asserting.

Improve this answer
3
  • 1
    The test wart is meaningful to Python's test framework. Which of course needn't change whether you're a fan of it, but does change the weight of argument needed to stop someone using it ;-)
    – Steve Jessop
    Jan 12, 2015 at 21:04
  • @SteveJessop Yeah, I keep forgetting that it's needed. When I was writing Python tests a while ago, I got into the habit of beginning them all with test_that_, like test_that_return_values_are_in_order and so on. Maybe a future version of the test framework will work around this requirement somehow.
    – Dawood ibn Kareem
    Jan 12, 2015 at 22:29
  • @DavidWallace: you can change the prefix if you want to.
    – Lie Ryan
    Jan 13, 2015 at 0:40
1

I'd leave these tests separate if that is how they've evolved to be. Yes test_order is likely to fail whenever test_length does, but you definitely know why.

I'd also agree that if the test_order appeared first and you found a testable failure was that the result was possibly not two values, adding that check as an assert and renaming the test to test_length_and_order makes sense.

If you had to also check the types of the values returned were integers, I'd include that in this "omnibus" results test.

But note now you have a battery of tests for the result of my_function(). If there are multiple contexts (or, more likely, multiple parameters) to test this can now be a subroutine to test all results from my_function().

However, when I write unit tests, I normally test the edge and bad cases separately from good input and normal output (partly because most of my "unit" tests are often mini integration tests), and often with multiple asserts, which are only broken into separate tests if they fail in ways where I find I want more information without debugging.

So I would probably start with your separate tests and expand test_length to test_length_and_types and leave test_order separate, assuming the latter is seen to be "normal processing".

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.