When writing (parameterized) tests for stateless functions, how do I determine the most ideal set of values to test with?

I'm writing a class with mathematical functions in C#, and I want to write unit tests for them. Today I want to write this function (really!):

int Clamp(int value, int lowerBound, int upperBound)
    // TODO: Implement.
    return 0;

As good TDD practice dictates, I'm writing the unit tests first and the implementation later. I can write a parameterized test for this stateless function, but have to decide on the best set of values to test with.

My current strategy for picking testing values is:

For each parameter, I can pick values from the whole set of 32-bit signed integers. I'm not using randomly generated values as that would make the test non-repeatable. To test the corner cases, I pick -1, 0, 1, but also the −2147483648 (minimum 32-bit signed integer) and 2147483647 (maximum 32-bit signed integer). I would also need to pick a 'normal' value, for example, 42, to make sure I'm not only testing corner cases. Is one normal value enough? And a negative one, let's say -63. Did I miss any?

−2147483648, -63, -1, 0, 1, 42, 2147483647

If any of the three parameters can have any of just these seven values, I'll end up with 343 tests for one function! Before I'm going to write 343 tests, I really have to ask:

Are these the right values to test with? In general, what strategy can I use to pick the most effective values to use?

  • 1
    Some of the frameworks out there allow you to specify multiple inputs for the same test. nunit.org/index.php?p=values&r=2.5 – CodeART May 1 '14 at 15:53
  • 1
    There's a technique for this called equivalence partitioning. Leaving this as a comment, because I don't know enough to make a good answer. – Andy Hunt May 1 '14 at 15:53
  • @AnduBursh That definitely sets up a good baseline. I would add that imperative code has a good chance of screwing up at the boundary between one equivalence class and another, so rather than picking 1 input that falls somewhere along the middle of its equivalence class, it'd be useful to pick the 2 inputs at the extremes of each class. – Doval May 1 '14 at 17:14
  • First select a test case that will trivially pass the test. Then add test cases that break with current implementation but require the least amount of work to implement. Keep adding tests and fixing the implementation until done. The result should be a set of tests that you could consider a spec for the function i.e. anyone with the test suite should be able to easily implement version of the function themselves. – lego Mar 19 at 10:27

You never said what Clamp() is supposed to do, so I'm assuming that it returns value, unless it is outside of the range, in which case it returns one of the two bounds.

I don't see any reason to think that -1, 0, or 1 are corner cases. They may often be corner cases, but there's no reason they'd act strangely in this function. If you want a 'normal' value, 42 or -63 works, but there is no need for both of them, unless you suspect that > and < don't work properly on negative numbers in C#. (I don't think you need to worry about that.)

So we could just use −2147483648, 'a normal value', and 2147483647. (We could even say that testing with the max/min integer values aren't really necessary. Presumably, C# > and < work up to the minimum and maximum; there isn't any danger of integer overflow.)

There are 6 permutations of 3 values, so we're down to 6 testcases. 6 testcases is not much, and we can easily just write them down and use them, but we don't know for certain that we've selected test cases that cover everything (all we've done so far is reduce the original set of test cases to something smaller).

If we want to be sure we've caught all the cases that matter, we could reduce the massively large set of input values (4 billion cubed) by partitioning them into equivalence classes. Then we only need 1 test per equivalence class, since the equivalence class would be defined as a set of inputs that all act alike.

The value of Clamp(a, b, c) depends on whether a is in the range, or above it, or below it. There should be 3 equivalence classes: [a < b and a < c], [a > b and a > c], and otherwise. The return value will be b, c, or a, respectively. This tells us not only what the tests should be, but how to write the code.

(There is one little thing that we haven't run into: what if the lower bounds is higher than the upper bounds. What I said in the previous paragraph applies if the assumption I made up at the top is right, but not if it isn't. It can be fixed easily, though, by swapping b and c or by returning Clamp(a, c, b) if b > c.)

  • Almost every paragraph you're making assumptions about the supposed implementation of Clamp(), which you and I don't know (since I haven't implemented it yet). For example, I might make a mistake with signed/unsigned comparisons, or I might distinguish between positive and negative values, and somehow completely forgot about 0. You can't make any assumptions about the implementation. However, I agree that I could specify what Clamp() is supposed to do, but I'm searching for a general way to determine the test values. My mathematical class will contain more functions than just Clamp(). – Daniel Pelsmaeker May 2 '14 at 0:15
  • The only place I make assumptions about implementation are the two paragraphs where I look at the original set of test cases, and all I assume there is that the implementation would use > or < (or possibly <= or >=, but that didn't seem important). Unless I was completely mistaken about what Clamp() should do, it's a reasonable assumption. If you can't make assumptions about the implementation, you can't split the input into equivalence classes, and you can't intelligently predict what the corner cases might be. – Michael Shaw May 2 '14 at 1:05
  • No doubt there are ways to implement Clamp() that would fail badly on some inputs -- but which inputs are problems depends on how it's implemented. The only way to be absolutely sure it works correctly, without any knowledge at all of how it's implemented, is to try every case. Doing some back-of-the-envelope estimation, running through every case for Clamp() would take more than 100 billion years. – Michael Shaw May 2 '14 at 1:13
  • You're mixing up implementation and specification. I can specify that Clamp() will return value when it is between lowerBound and upperBound, lowerBound when it is less than lowerBound, and upperBound when it is greater than upperBound, an throws an exception when upperBound is less than lowerBound. Without assuming anything about how I'm going to implement this, how can I pick the most convenient values? And I'm looking for a strategy for any stateless function in general that you know the specification but not the implementation of. – Daniel Pelsmaeker May 2 '14 at 11:25

Test Driven Development principles suggest that you test cases cover a range of conditions, not only testing over a range of valid input, but testing for handling of error conditions as well.

So first start testing how it handles invalid / valid input, to see if it correctly handles erroneous input,

That includes:

  • Simple known good case (PASS)
  • Simple known "bad" input (FAIL)
  • Based on the underlying datatype, values at boundaries for that datatype (e.g. INT_MAX, INT_MIN, a string of maximum or minimal length including the empty string)
  • With numerical values, testing a signed value
  • Testing values over a complete range of input values, beyond what makes sense for the application usage. If you assumes the parameter is normally going to be between 1 and 10, how does it handle 1000000 or -10? This can be tricky to comprehensively test for floating point in particular.

The extensiveness of testing is determined on factors such as how widespread usage is (i.e. is it an one-off usage at the moment versus inclusion in a common library), how critical are failures? (user loses a casual game versus vehicle's brakes cease to function)

If any of the three parameters can have any of just these seven values, I'll end up with 343 tests for one function! Before I'm going to write 343 tests,

Don't confused tests with test cases, test cases should be thought of as mere data.

You should write code that loops through those test cases that may be stored as external CSV text files, and see if they match the expected outcome. Break the cases into lists of "good" (expect to pass) values and "bad" (expect to fail) values and simply test whether the function behaves (PASS/FAIL) as expected give the combination of input data (all cases with valid input should pass, all cases with any invalid input should fail), without verifying the correctness of the output.

Then for a set or subset of interesting input values you can compare the output values to the expected output values, that are normally generated independently of the code being developed (e.g. for many simpler cases for numeric calculations a spreadsheet can be used), making sure to handle floating point comparisons sanely.

If you generate the test case data automatically, then the number of test cases only becomes a factor once the execution speed is an issue, an increasingly rare issue when testing at function test.

In the case of your example function, where for any input the output should be valid, then you can simple test the correctness of the output given the particular combination of input values.

It does help force you to limit the numbers of parameters to a function, which is normally a good thing as it limits the functions to do one thing well, and not become a ugly monster of special cases based on those additional values.

  • While 343 test cases seems like a lot, I'm not saying it is too much or not feasible. Indeed, automation is the way to go here. But the question was more along the lines of: what strategy can I use to pick the best values for my test cases? – Daniel Pelsmaeker May 1 '14 at 17:13
  • Values which represent common modes of failure. The frequency or likelihood of these failure modes can be weighted against the important of testing (i.e. how wide spread is usage & severity of failure) to give a pragmatic sense of how through or exhaustive you need to be. The other criteria often used is how much time is available for you to spend writing and validating the tests for that function. In the end it comes down to management of risk analysis and time. – mctylr May 3 '14 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.