Is it possible to test if/else trees properly without coding to the implementation?

Question

I have been writing tests for a lot of long if/else trees recently, and I'm finding it a little discouraging. I want to speak in concrete terms, so consider the following example (I'll write in Ruby syntax but the example is quite general):

A user can either be an admin? or not (boolean method). A file can either be public? or not. An admin can open any file, and a public file can be opened by any user. A non-admin user cannot open a non-public file. To define whether a user can open a file, we make the following checks:

if user.admin?              if file.public?
  true                        true
elsif file.public?          elsif user.admin?
  true                        true
else                        else
  false                       false
end                         end

I'll call the left implementation "admin-first", and the right one "file-first". Clearly, both implementations return the same values for every possibility.

I want to write a test for this block of code, which tests the behaviour and not the implementation. In particular, there are three test cases I care about:

• Given an admin user, and any file whatsoever, the block returns true.
• Given any user whatsoever, and a public file, the block returns true.
• Given a non-admin, non-public file, the block returns false.

My problem is that I cannot see a way to actually write tests which mean the above conditions.

One option ("maximum correctness") is to test all four options of true/false. Of course, this is a toy example and the cases I've been looking at have upwards of 15 conditions which can (in principle!) vary. Even with only four conditions, which is relatively common, this approach is impractical. It's also very ugly, especially in the extreme case: Why do I have to write more tests than there are distinct behaviours (in some cases, many many more)?

A similar option ("randomness") is choosing inputs randomly, but random tests are generally discouraged, and I believe the arguments against it.

A pragmatic approach when faced with the "admin-first" implementation is to test the following three cases instead:

• Provide an admin user (any file will do, because the tree will short-circuit).
• Provide a non-admin user with a public file.
• Provide a non-admin user with a non-public file.

I dislike this for several reasons. First, the behaviour case of "any user with a public file" is not checked. That's an aesthetic point really, but it also makes the tests confusing ("why is this property set? is it relevant?"). Second, the test will fail if I were to rearrange (preserving behaviour!) into the "file-first" implementation, so you're forced to actually write the tests to go along with the code, or try out every possiblity as before. As you go down the cases, the number of irrelevant bits of data you need to specify only gets bigger.

The tests can be made to pass even after reordering if we change the first check to specify that the file is non-public, rather than unspecifed. Again, I'm not satisfied:

1. Now two of the three test cases fail to test the behaviour.
2. In the case of more branches, you need to specify every option of data at every level of the tree, which is just plain awful.
3. It is "intuitively obvious" that the overlap case (admin and public) does not need to be tested if the other two code paths work correctly. But that's only true if you know what the code looks like! In more complex cases, this inference is less clear and more confusing for a test-reader.

For the case of very many branches, you might say something like "refactor it to hide some of those booleans inside of other methods", but unfortunately I am writing these tests precisely because I want to refactor this code. The tests have to come first.

The two things I thought of first, namely using guard clauses or chains of && or ||, have the same problem, because of short-circuiting.

So, my question: Is there any way to test code resembling this, or to restructure code that looks like this, in such a way that we can really test behaviour and not implementation?

Answer 1

You have to test all the scenarios which are defined in your acceptance criteria.

When you have an array of options like this I would use Data driven tests

[TestCase("admin","public",true)]
[TestCase("normal","public",true)]
[TestCase("admin","private",true)]
[TestCase("normal","private",false)]
public void CanFileOpen(user,file,expected)
{
     var actual = file.Open(user);
     Assert.AreEqual(actual, expected);
}

If you have inputs which are numeric rather than have a set of possible values then you usually test some known results plus the minimum and maximum possible values, 0, null etc to check for edge cases.

Answer 2

Brevity eludes me. I was commenting on randomized data driven tests and just wanted to keep going.

---

For the case at hand, with only four possible input points. I agree on enumerating all posilibities. However, this is a simple case.

I really want to point out that randomized data driven tests are useful. They have the potential to express the behaviour of complex system better than unit tests. If tests as documentation is a goal, that is good. It means you can look at the tests and know what to expect from the system. Plus, it is documentation that is tested. No chance of mismatching the real system.

To be clear, I am not suggesting to make one single test considering all possible input per exposed API. Will come back to that.

---

We need to be clear that randomized data they are not unit tests. Also there is poor support for them in automatic testing frameworks... because most of those frameworks were created for unit testing first.

Common argument against them is lack of repeatability, we need a testing system that will remember, document and reproduce found failing conditions. Not having a good framework could be a reasonable reason to not use such tests.

---

We have the idea that if all tests passes the system is OK. This implies a methodology that guarantees that we have written enough tests to capture the behavior of the system. Otherwise, all tests passing could mean we have not done enough tests to find when the system fails. The methodology that I would suggest to get that, when possible, is London style test driven design.

Please notice "when possible". Sometimes it is not. My pet-peeve is testing a threading primitive. Most test will have an inherent race condition, and then to avoid having the testing framework hang (which is a bad thing™), we introduce a timeout. Now we have moved from unit tests to acceptance tests. This why the common advice for creating such threading code is to base it on first principles... but enough about that.

Your tests characterize points in the input space, if the input space is large enough (say, all integers or all floats), you cannot write a test for each possible value... well you could try, with some code generation, I guess. However, notice that the input space could be virtually infinite, for example, the set of all possible strings.

There is a person in the mix, there must be. Somebody conceptualizes a behavior over an input space from some data points that are being tested. There is value in expressing these tests in more abstract terms. If I am testing an identity function, I can only test that it returns what I input for limited set of values. If a human picking a value at random that is good enough for unit testing, then having the system pick a value at random should be good enough too.

---

Back to how to approach the situation at hand with randomized data driven tests... you say:

An admin can open any file

That is a test. You fix that the user in the test is an admin, and you let the file be random. After all, the file should not matter.

a public file can be opened by any user

Another test. Now you fix that the file is public and let the user be random.

A non-admin user cannot open a non-public file

One more test. No random in this one. We can say that this is non-randomized test... then we can say that randomized tests complement non-randomized ones.

By the way, you would tell the system to test some number of combinations of values. Thanks to that, yes, this is enough.

More importantly, if you are following a test driven design methodology, you will have to figure out what is the simplest way to pass these failing tests. And that will lead to the correct implementation.

And it will lead there nicely because of the close match between requirements and tests. If I hard-code inputs the simplest implementation that passes the tests could be to just return the right value for the hard-coded inputs, until I hard-code enough inputs... and for the case at hand, I strongly believe that is hard-coding all combinations.

---

Of course, the fear with using random is that I could be in a situation where all tests passed, yes the system is not OK, because by chance the values picked by the tests were ones such that they avoided a latent bug.

The same is true if you, the human, are picking the values you put in your unit tests. All tests could be passing, and there could be a latent bug not found because it does not happen with input you picked.

In this regard, randomized data driven tests have the upper hand: they could, by chance, discover the bug. We, of course, need to not lose track of the values used in the randomized test so we can reproduce it... which also works as a regression test.

You, of course, will avoid failing to characterize the behavior of the system in your tests if you have a good methodology (and you are working in a problem domain where it is possible). Well, the same methodology can work with randomized data driven tests. And these tests will be more succinct and express the behavior of the system – for the benefit of the humans reading it – in cleaner tests (assuming you have a good framework for this kind of tests).