Unit Testing a Function which produces "limit less" results

Question

I am pretty new-ish to unit testing in a more nuanced sense. If have an application function which based on users given input could produce an seeming "limit-less"combination of output, whats the proper way to unit test this and how far might testing go?

I am thinking of something like building an XML file which given input you can create an unlimited combination of files. Producing the individual functions for things like root nodes, child nodes, etc, and testing them is pretty straight forward to me, but if you wanted to take all those functions and build the file, how might one test those combinations?

Answer 1

There are two types of testing: black box testing and white box testing.

When you do black box testing, you don't know the internals of the code you test: the only thing you know is the interface and what the code is expected to do.

In white box testing, you can see how the code actually performs the task you are about to test.

Unit testing is white box testing. This means that you are doing unit testing based not on the eventually unlimited number of inputs and outputs, but on the branches, that is the possible ways to execute code.

Let's take the following code as an example:

string sayHello(string name) {
    if (name == null) {
        return "Hello!";
    }

    return "Hello, {0}!".format(name);
}

It does take a parameter which accepts a nearly unlimited number of values and leads to a nearly unlimited number of results, and it would be probably quite problematic to test every combination. Instead, you can explore the code and test it based on the logic of the code itself:

1. Is the result of sayHello("Jeff") actually "Hello, Jeff!"?

2. What about sayHello(null)? Is it "Hello!"?

Once you tested those cases, you are quite sure, by looking at the code, that it will work as well if we pass to it "Mary" or "Stephen". Thus, no need to test all the names you can imagine.

Another things you should take in account are:

• The requirements. For instance, you may wonder what would be the output of the method for an empty string. Does it make sense to display "Hello, !" to the user?

• The language, framework and system constraints. What if my name is actually 2^31 - 1 characters long? What would happen when the method will call format, attempting to produce a string which is 2^31 + 7 characters long?

Answer 2

The goal of unit testing is not to test all possibilities of input versus output.

The goal of unit testing is to test each and every business rule, and within the scope of each business rule, all the potentially problematic cases. (Also known as "edge cases".)

So, basically, if your user gives a specific, known-in-advance piece of input, (which has been hard-coded into the test,) does your code yield a specific, known-in-advance piece of output (also hard-coded into the test) ?

And, if your user gives an empty string, does your code behave?

And, if your requirements say that your user must be able to type up to 10,000 characters, does your code in fact handle 10,000 characters? And does it fail gracefully (with a proper exception, which would later be translated to an error message to the user) if the user supplies 10,001 characters?

Also, code coverage is a measurement of how good your unit testing is, not in the sense that if you have 100% code coverage then you are good to go, but in the sense that if you have significantly less than 100% code coverage then it is possible that you have missed something. So, use a code coverage tool to see what parts of your code are not covered by your tests, and if any of them is crucial, then construct a test case in such a way that this piece of code will be covered.

Answer 3

The general term for this type of testing is combinatorial testing.

One commonly used technique for combinatorial testing is all-pairs or pairwise testing, which cuts down on the combinatorial explosion of test cases by trying to find a smaller 'interesting' set of cases. With the example of generating XML files with a typical schema, a naive 'try every option in combination' approach would leads to literally millions or billions of cases. Pairwise testing can probably identify a few tens of XML files that exercise each combination of schema features.

Another is purely random or generative testing, which avoids the effort required to boil the test cases down to a minimum set by simply generating a few thousand at random and relying on the law of large numbers.

Testing of this kind might well be considered not to be unit testing. In fact, it is commonly done at a higher level, perhaps by an external test team. However, if it is done by the author of the code and at a detailed technical level, then in practice there is not much distinction.

Some (Java-based) tools for this low-level approach to combinatorial testing are jcunit, junit-quickcheck, and junit-theory-suite. Most other languages have similar tools available.