0

Suppose I'm writing a calculator engine with a class as follows...

public class ArithmeticExpression
{
    ArithmeticExpression Add(double operand) { /* ... */ }
    ArithmeticExpression Subtract(double operand) { /* ... */ }
    // ... other useful operations like division, multiplication, etc.
    double ProduceResult() { /* ... */ }
} 

... and the goal is to defer evaluation of the expression until ProduceResullt() is called, so that correct order of all stacked operations is maintained.

My question is:

How should I tackle with unit testing this class without getting overwhelmed by the number of combinations of method calls?

It is clear to me that the order of calls should be tested somehow, but writing tests like this:

public void Multiply_CalledAfterAdd_TakesPrecedenceOverAdd()
{ 
    // ...
}
public void Multiply_CalledAfterDivide_EvaluatesLeftToRight()
{
    // ...
}

can get maddening and out of control quite quickly. And it's not even about those 16 tests for the four basic arithmetic functions. Suppose in the future I decide to extend the calculator with a modulo operation - that's 9 almost identical tests more to begin with.

On the other hand - the whole puprose of the class is to keep the order of operations maintained, so it's got to be verified somehow.

Wrapping up - I know the example is trivial, but it's a general problem that I struggle to find an answer to. Any help is appreciated, thanks.

2
  • I'd recommend doing some reading on Combinatoric and PairWise testing strategies
    – user172618
    Jul 23, 2019 at 18:48
  • If I understand it correctly, pair-wise testing is truly useful when the number of variables is relatively high compared to the number of possible values each of them can have, which is hardly the case here. It's an interesting related topic, though. Thanks for sharing.
    – mdx
    Jul 23, 2019 at 19:46

3 Answers 3

1

It's impossible to test all possible combinations, so your tests only need to cover a well-chosen subset.

For example, if you know that all operations are implemented similarly (with the only difference being the operator) you can test the correct ProduceResult() sequencing using a sequence of operations which wouldn't yield the correct result if performed in the wrong order. If you have methods for parenthesized expressions and operator precedence you need some more cases.

Of course, you should have tests for each single operation to achieve high coverage and confidence that they work correctly even in border cases.

For the sequencing, just a few cases that test correct results and correct propagation of intermediate errors (division by zero, overflow, injection of NaNs) should be enough.

2

Problems like this, I tend to go a little white-box, and make sure I am covering every line of code and every condition. Assuming you are using some variant of the shunting-yard algorithm that has an operator stack, I would test combinations like:

  • operator stack is empty
  • operator stack has a higher-precedence operator
  • operator stack has a lower-precedence operator
  • operator stack has an equal-precedence operator
  • operator stack has a left paren at the top
  • operator stack has a left paren buried under other operators

Take every combination with the following values for the next token:

  • a number
  • a left paren
  • a right paren
  • an operator
  • the end of input

Multiply the 6 operator stack states with the 5 next token states, and you get a total of 30 tests. Not intractable, and very likely to work with any arbitrary input. If despite all that you find a bug during manual testing, add a specific test for that bug, so you won't accidentally break it later.

1
  • I eventually ended up doing it this way, but only because in this case the number of tests is not that intimidating, just as you mentioned. However, were I to write 300 nearly identical tests instead of 30, I'd honestly think about skipping the most trivial ones and focusing on what's really complicated (logic-wise), following what Hans-Martin Mosner wrote.
    – mdx
    Jul 26, 2019 at 17:47
1

Your problem is a small, specific example of a larger, more general one: how do you account for every possible permutation of a function or class? And the answer, of course, is "You don't."

Consider this trivial case:

int sum(int a, int b)
{
    return a + b;
}

Why don't we test every possible combination of parameters to this function? For a number of reasons. First, we are fairly confident that math works as it should in any decent programming language. Second, because it's not practical. And third, because it is only of marginal benefit.

So, if we don't test every possible way that our code can execute, then what do we test? And the answer is: enough to be confident that the code works as it should.


If you find yourself writing a lot of tests like this for your code, your time might be better spent finding a way to simplify the code under test so that the permutations are reduced.

I worked on a security system recently that used a lot of if/else ladders to accomplish its task. If/else ladders are not only very verbose, they are also difficult to understand beyond a trivial size. Changes sometimes required adding an entirely new block of if's to override old ones. Exhaustive testing is almost mandatory.

I replaced these with boolean expressions at single endpoints in a View Model. The testing required for those amounts to visual inspection and runtime verification.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.