3

I am test driving a method which is parameterized by two inputs and depends on a Strategy. Here's what I have:

[TestCase(input_a1, input_b1, strategy1)]
[TestCase(input_a2, input_b1, strategy1)]
[TestCase(input_a3, input_b1, strategy1)]
[TestCase(input_a1, input_b2, strategy1)]
[TestCase(input_a2, input_b2, strategy1)]
[TestCase(input_a3, input_b2, strategy1)]

[TestCase(input_a1, input_b1, strategy2)]
[TestCase(input_a2, input_b1, strategy2)]
[TestCase(input_a3, input_b1, strategy2)]
[TestCase(input_a1, input_b2, strategy2)]
[TestCase(input_a2, input_b2, strategy2)]
[TestCase(input_a3, input_b2, strategy2)]

My test method looks like this:

[TestCase(expected, input_a3, input_b2, strategy2)]
void TestMethod(Output expected, Input a, Input b, Strategy strategy)
{
    var TestObject = new TestObject(strategy);
    var actual = testObject.TestMethod(a, b);
    // Assert expected == actual
}

It's not hard to imagine how I got here. I started by varying input_a and keeping input_b and strategy1 constant, ending up with 3 test cases.

[TestCase(input_a1, input_b1, strategy1)]
[TestCase(input_a2, input_b1, strategy1)]
[TestCase(input_a3, input_b1, strategy1)]

Then I copied those test cases and varied input_b with the other two constant, multiplying my test cases.

[TestCase(input_a1, input_b2, strategy1)]
[TestCase(input_a2, input_b2, strategy1)]
[TestCase(input_a3, input_b2, strategy1)]

Then I varied the strategy, again multiplying my test cases.

[TestCase(input_a1, input_b1, strategy2)]
[TestCase(input_a2, input_b1, strategy2)]
[TestCase(input_a3, input_b1, strategy2)]
[TestCase(input_a1, input_b2, strategy2)]
[TestCase(input_a2, input_b2, strategy2)]
[TestCase(input_a3, input_b2, strategy2)]

I feel that my test cases are growing at an alarming rate. Now I need to add a third Strategy. If I continue like this, I would be adding another set of test cases. If I added another input, input_c, I would multiply my sets of test cases by at least 2.

Is it too verbose to test every combination of input_a, input_b, and strategy?

  • 4
    If only there was a way to get a computer to repeat things a certain number of times with predictable changes to variables at certain intervals. – JeffO Feb 27 '18 at 13:38
  • @JeffO Never heard of such a thing! It would actually help me with my current assignement! /s – Euphoric Feb 27 '18 at 15:27
  • 1
    Short answer: instead of generating all possible combinations, learn an old, well-known testing technique called "Equivalence partitioning" for reducing the number of necessary test cases. – Doc Brown Feb 27 '18 at 20:17
  • @Euphoric - there's probably some type of for/loop in your language of choice for running tests. – JeffO Feb 28 '18 at 14:22
6

You can use NUnit's combinatorial option like this

void TestMethod(
    [Values(input_a1, input_a2, input_a3)]Input a,
    [Values(input_b1, input_b2)]          Input b,
    [Values(strategy1, strategy2)]        Strategy strategy)
{
    var TestObject = new TestObject(strategy);
    var actual = testObject.TestMethod(a, b);
    // Assert
}

This will result in 3*2*2 = 12 total test cases, which are cross-join of all given values.

As for quickly growing number of tests. While I think it smells, it is not a huge problem. As long as all the test cases run fast, there is no problem generating thousands of test cases.

  • Why do you think it smells? Why wouldn’t you attempt to test all the edges? I want to upvote (neat feature I didn’t know about), but that comment makes me hesitate. – RubberDuck Feb 27 '18 at 10:38
  • 3
    @RubberDuck I feel this kind of cross-joined test cases make it too easy to create impression of having lots of tests. Also, cross-join cases can sometimes be unnecessary. It migh be possible to properly test single input at a time. Or just pick appropriate test cases instead of generating all possible permutations. But it being smell, I don't really have solid reason, for it. Just bits of experience. – Euphoric Feb 27 '18 at 11:16
  • Fair perspective. – RubberDuck Feb 27 '18 at 11:31
  • I don't think I can do this, because I need to assert that an expected value is equal to the actual value for each test case. I'm sorry I didn't include this information before. In reality my TestMethod() takes in an expected variable as well for each test case, which I don't think I can (or should) calculate programatically without reimplementing the method I'm testing. – Giuliano Conte Feb 27 '18 at 14:49
  • 1
    @GiulianoConte In that case, I don't think there is viable shortcut. You should enumerate as many cases as to give yourself confidence that the parameter space if sufficiently covered. – Euphoric Feb 27 '18 at 15:29
1

First, this is the exact scenario that many people use scripting for. In my particular case, I usually use Tcl scripts to generate unit tests for C code, but the principle broadly applies. Once you have a suitable script in place, adding additional parameters to generate further tests for a given function is a trivial exercise. Functionally, this is exactly the same thing as using a parameterized test case definition from your unit test framework of choice, so if your framework provides that ability, by all means use it.

Second, for a function that has several parameters, you'll want to examine the unit test structure to make sure that you touch the normal case and any edge or special cases, but, you only need to touch the normal case once. I've seen a lot of unit tests where the author supplied several values from the normal range of values (one to a test), thinking that they were somehow increasing the effectiveness of the test.

If that line of reasoning held true, then the perfect set of unit tests for a function that takes a signed 32-bit integer would have ~ 2 billion variations. Clearly, that is not the case. Perversely, you have to watch out for this even more carefully when you are using scripts to generate the test functions, because it is so easy to do. When you are deciding which values to pass to your unit tests, remember, "Test what you have to, but only what you have to."

Third, for combinations of parameters, the same considerations apply. As the number of parameters to a function increases, then the number of unit tests for that function is going to unavoidably increase. Yes, it's going to get lengthy (and tedious if you have to write them all out by hand). That doesn't mean that you shouldn't have those tests; it just means that you should dump all of that tedium off onto the computer if at all possible.

0

Since you want all combinations, why not generate the test cases programmatically?

generateTestCases()
{
    a_inputs = {input_a1, input_a2, input_a3};
    b_inputs = {input_b1, input_b2, input_b3};
    strategies = {strategy1, strategy2, strategy3};

    foreach(a_input in a_inputs)
    {
        foreach(b_input in b_inputs)
        {
            foreach(strategy in strategies)
            {
                // create/run test case on a_input, b_input, strategy
            }
        }
    }
}

This will run every combination of inputs and strategies. Adding more inputs or strategies only requires changing one line. This function can either generate the list of TestCases which are then read by the testing methods, or the testing methods can be run from inside the innermost loop above, whichever is better for your code base.

  • While this will run all combinations, the test framework may only see a single test case. That makes test failures difficult to debug – which combinations led to the failure? So while this is a legitimate approach, it should only be used if the framework doesn't offer a more convenient way to run parameterised tests. – amon Feb 27 '18 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.