25

Is it better to have either a

  • Deterministic test suite, that results in the same tests succeeding
  • Non-deterministic test suite, which potentially possibly covers more cases

?

Example: You write a test suite to test controller functionality in an MVC application. The controller requires application data from a database as input during the test. There are two options to do this:

  • You hardcode which row(s) from the test database are selected as input (e.g. the 10th and 412th row)
  • You use a random number generator to pseudorandomly pick the data from the database (two rows selected by a random number generator)

The first is deterministic: every run of the test for the same revision of code should yield the same result. The second is non-deterministic: every run of the test suite has the possibility to yield a different result. The randomly picked data might however be a better representation of data edge cases. It might simulate a user feeding our controllers with unpredictable data better?

What are reasons to choose one over the other?

7
  • 6
    That test just fails, sometimes. martinfowler.com/articles/nonDeterminism.html
    – user40980
    Dec 17 '13 at 13:54
  • 2
    Thanks for that link. With that article in mind, I felt I needed to clarify that non-determinism means in the context of this test suite. Because data is selected randomly from a database, all data fed to the controller is valid data by default. This means that false negatives don't exist in the test suite when it comes to the non determinism. In a way, this randomness simulates a user selecting data 'at random' for use in a controller. This is not necessarily the same non-determinism the article discusses, right?
    – DCKing
    Dec 17 '13 at 14:15
  • 3
    recommended reading: Why is asking a question on “best practice” a bad thing?
    – gnat
    Dec 17 '13 at 14:19
  • 13
    @DCKing: Consider what happens if your test fails. Okay, you have a bug. Uh, now what? Run it again in debug mode! Where it succeeds! Like it does the next hundred times you run it, and then you write off the issue as a cosmic ray strike. Non-determinisim in tests sounds absolutely unworkable. If you feel the need to cover more ground in your test cases, cover more ground. Initilise your RNG with a set seed and run the "test" a few hundred times with consistently random values.
    – Phoshi
    Dec 17 '13 at 14:35
  • 2
    (finally got around to a machine where I could properly search twitter - the "That test just fails sometimes" is from the #FiveWordTechHorrors on Twitter - wanted to properly credit it)
    – user40980
    Dec 17 '13 at 15:25
39

When every run of the test suite gives you the possibility to yield a different result, the test is almost completely worthless - when the suite shows you a bug, you have a high chance that you cannot reproduce it, and when you try to fix the bug, you cannot verify wether your fix works (or not).

So when you think you need to use some kind of random number generator for generating of your test data, either make sure you always initialize the generator with the same seed, or persist your random test data in a file before feeding it into your test, so you can re-run the test again with exact the same data from the run before. This way, you can transform any non-deterministic test into a deterministic one.

EDIT: Using a random number generator to pick some test data is IMHO sometimes a sign for being too lazy about picking good test data. Instead of throwing 100,000 randomly choosen test values and hope that this will be enough to discover all serious bugs by chance, better use your brain, pick 10 to 20 "interesting" cases, and use them for the test suite. This will not only result in a better quality of your tests, but also in a much higher performance of the suite.

8
  • Thanks for your answer. What is your opinion on the comment I made to my question?
    – DCKing
    Dec 17 '13 at 14:18
  • 1
    @DCKing: if your really think a random generator will be better in picking good test cases than you (what I doubt), use it once to find combinations of test data where your program fails, and put those combinations into the "hardcoded" part of your test suite.
    – Doc Brown
    Dec 17 '13 at 14:39
  • Thanks again. Updated my answer so that it doesn't seem to apply to just MVC apps.
    – DCKing
    Dec 17 '13 at 16:07
  • 1
    In some UI contexts (for instance, games taking controller input) having test programs that generate random key entry can be useful for stress testing. They can uncover defects that are hard to find with deliberate input. Dec 17 '13 at 18:56
  • 1
    What if when your random test fails, it outputs the fail scenario, which you can then use to further troubleshoot the problem? This is the case with property-based testing in frameworks like QuickCheck (Haskell) and ScalaCheck (Scala).
    – Andres F.
    Dec 12 '16 at 14:30
10

Both.

Deterministic and nondeterministic tests have different use cases and different values to your suite. Generally nondeterministic can't provide the same precision as deterministic testing, which has slowly grown into "nondeterministic testing provides no value." This is false. They may be less precise, but they can also be much broader, which has its own benefits.

Let's take an example: you write a function that sorts a list of integers. What would be some of the deterministic unit tests you'd find useful?

  • An empty list
  • A list with just one element
  • A list with all of the same element
  • A list with multiple unique elements
  • A list with multiple elements, some of which are duplicates
  • A list with NaN, INT_MIN, and INT_MAX
  • A list that's already partially sorted
  • A list with 10,000,000 elements

And that's just a sorting function! Sure, you could argue that some of these are unnecessary, or that some of these can be ruled out with informal reasoning. But we're engineers and we've seen informal reasoning blow up in our face. We know we're not smart enough to completely understand the systems we've built or fully keep the complexity in our heads. That's why we write tests in the first place. Adding nondeterministic testing just says that we might not necessarily be smart enough to know all of the good tests a priori. By throwing semi-random data into your function, you're much more likely to find an edge case you missed.

Of course, that doesn't rule out deterministic testing either. Nondeterministic testing helps find bugs in huge swaths of the program. Once you've found the bugs, though, you need a reproducible way to show that you fixed it. So:

  • Use nondeterministic tests to find bugs in your code.
  • Use deterministic tests to verify fixes in your code.

Note that this means a lot of solid advice about unit tests don't necessarily apply to nondeterministic tests. For example, that they must be fast. Low-level property tests should be fast, but a nondeterministic test like "simulate a user randomly clicking buttons on your website and make sure you never get a 500 error" should favor comprehensiveness over speed. Just have a test like that run independently of your build process so that it's not slowing down development. For example, run it on its own private staging box.

6
  • Actually, I don't understand why non-deterministic tests should have any benefits over tests with pseudo-random, but deterministic test data.
    – Doc Brown
    Jun 24 '21 at 13:40
  • 1
    @DocBrown are you familiar with fuzzing?
    – Hovercouch
    Jun 25 '21 at 4:19
  • @DocBrown Well if you run it once you know that case works. Running fifty different test runs is more likely to find a new problem, than running the same test fifty times.
    – user253751
    Jun 29 '21 at 8:50
  • @user253751: still one has to make sure in case a problem occurs, one can fix it and run the same test again which failed before. So there must be some precaution taken to make things deterministic again.
    – Doc Brown
    Jun 29 '21 at 9:36
  • @DocBrown Most generative testing tools provide the run seed and inputs that failed, so you can either rerun it or extract it to an example test. More sophisticated tools, like Hypothesis, also keep a failed input database for later auditing and reuse.
    – Hovercouch
    Jun 29 '21 at 17:28
5

Both deterministic and non-deterministic have a place

I would divide them as follows:

Unit tests.

These should have deterministic, repeatable tests with the the exact same data every time. Unit tests accompany specific, isolated code sections and should test them in a deterministic fashion.

Functional and input stress tests.

These can use the non-deterministic approach with the following caveats:

  • that fact is clearly delineated and called out
  • the random values selected are logged and can be re-tried manually
0

Random unit tests are good, but can miss corner or edge cases, which should be tested deterministically. Generally, random unit tests should have an invariant.

I've written a set of tests for a matrix arithmetic module. One test makes a matrix with all ones on the diagonal, random numbers on one side, and zeros on the other. The determinant of this is guaranteed to be 1; that's the invariant. Then it permutes the rows and columns with an even permutation; the determinant is still 1. Then it calculates the determinant by calling the module. The row operations multiply rows by random numbers, introducing roundoff error, but the result is still close to 1.

Another test calls a function which returns the intersection type of two line segments. The edge case of three of the endpoints being collinear is unlikely to result from random choice, so I test it deterministically.

-1

You don't really want deterministic vs. non-deterministic.

What you might want is "always the same" vs. "not always the same".

For example, you might have a build number which increases with each build, and when you want some random numbers, you initialise a random number generator with the build number as seed. So every build, you do your tests with different values, giving you more chances to find bugs.

But once a bug is found, all you need to do is run the test with the same build number, and it is reproducible.

1
  • 2
    Or if you don't have a build number to use, place the initial value of the seed in the output of the test run, so you can again re-run tests with the same seed. Dec 12 '16 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.