How do you structure test cases on objects with multiple degrees of freedom?

Question

I am test driving a method which is parameterized by two inputs and depends on a Strategy. Here's what I have:

[TestCase(input_a1, input_b1, strategy1)]
[TestCase(input_a2, input_b1, strategy1)]
[TestCase(input_a3, input_b1, strategy1)]
[TestCase(input_a1, input_b2, strategy1)]
[TestCase(input_a2, input_b2, strategy1)]
[TestCase(input_a3, input_b2, strategy1)]

[TestCase(input_a1, input_b1, strategy2)]
[TestCase(input_a2, input_b1, strategy2)]
[TestCase(input_a3, input_b1, strategy2)]
[TestCase(input_a1, input_b2, strategy2)]
[TestCase(input_a2, input_b2, strategy2)]
[TestCase(input_a3, input_b2, strategy2)]

My test method looks like this:

[TestCase(expected, input_a3, input_b2, strategy2)]
void TestMethod(Output expected, Input a, Input b, Strategy strategy)
{
    var TestObject = new TestObject(strategy);
    var actual = testObject.TestMethod(a, b);
    // Assert expected == actual
}

It's not hard to imagine how I got here. I started by varying input_a and keeping input_b and strategy1 constant, ending up with 3 test cases.

[TestCase(input_a1, input_b1, strategy1)]
[TestCase(input_a2, input_b1, strategy1)]
[TestCase(input_a3, input_b1, strategy1)]

Then I copied those test cases and varied input_b with the other two constant, multiplying my test cases.

[TestCase(input_a1, input_b2, strategy1)]
[TestCase(input_a2, input_b2, strategy1)]
[TestCase(input_a3, input_b2, strategy1)]

Then I varied the strategy, again multiplying my test cases.

[TestCase(input_a1, input_b1, strategy2)]
[TestCase(input_a2, input_b1, strategy2)]
[TestCase(input_a3, input_b1, strategy2)]
[TestCase(input_a1, input_b2, strategy2)]
[TestCase(input_a2, input_b2, strategy2)]
[TestCase(input_a3, input_b2, strategy2)]

I feel that my test cases are growing at an alarming rate. Now I need to add a third Strategy. If I continue like this, I would be adding another set of test cases. If I added another input, input_c, I would multiply my sets of test cases by at least 2.

Is it too verbose to test every combination of input_a, input_b, and strategy?

Answer 1

You can use NUnit's combinatorial option like this

void TestMethod(
    [Values(input_a1, input_a2, input_a3)]Input a,
    [Values(input_b1, input_b2)]          Input b,
    [Values(strategy1, strategy2)]        Strategy strategy)
{
    var TestObject = new TestObject(strategy);
    var actual = testObject.TestMethod(a, b);
    // Assert
}

This will result in 3*2*2 = 12 total test cases, which are cross-join of all given values.

As for quickly growing number of tests. While I think it smells, it is not a huge problem. As long as all the test cases run fast, there is no problem generating thousands of test cases.

Answer 2

First, this is the exact scenario that many people use scripting for. In my particular case, I usually use Tcl scripts to generate unit tests for C code, but the principle broadly applies. Once you have a suitable script in place, adding additional parameters to generate further tests for a given function is a trivial exercise. Functionally, this is exactly the same thing as using a parameterized test case definition from your unit test framework of choice, so if your framework provides that ability, by all means use it.

Second, for a function that has several parameters, you'll want to examine the unit test structure to make sure that you touch the normal case and any edge or special cases, but, you only need to touch the normal case once. I've seen a lot of unit tests where the author supplied several values from the normal range of values (one to a test), thinking that they were somehow increasing the effectiveness of the test.

If that line of reasoning held true, then the perfect set of unit tests for a function that takes a signed 32-bit integer would have ~ 2 billion variations. Clearly, that is not the case. Perversely, you have to watch out for this even more carefully when you are using scripts to generate the test functions, because it is so easy to do. When you are deciding which values to pass to your unit tests, remember, "Test what you have to, but only what you have to."

Third, for combinations of parameters, the same considerations apply. As the number of parameters to a function increases, then the number of unit tests for that function is going to unavoidably increase. Yes, it's going to get lengthy (and tedious if you have to write them all out by hand). That doesn't mean that you shouldn't have those tests; it just means that you should dump all of that tedium off onto the computer if at all possible.

Answer 3

Since you want all combinations, why not generate the test cases programmatically?

generateTestCases()
{
    a_inputs = {input_a1, input_a2, input_a3};
    b_inputs = {input_b1, input_b2, input_b3};
    strategies = {strategy1, strategy2, strategy3};

    foreach(a_input in a_inputs)
    {
        foreach(b_input in b_inputs)
        {
            foreach(strategy in strategies)
            {
                // create/run test case on a_input, b_input, strategy
            }
        }
    }
}

This will run every combination of inputs and strategies. Adding more inputs or strategies only requires changing one line. This function can either generate the list of TestCases which are then read by the testing methods, or the testing methods can be run from inside the innermost loop above, whichever is better for your code base.