27

Is it better to have either a

  • Deterministic test suite, that results in the same tests succeeding
  • Non-deterministic test suite, which potentially possibly covers more cases

?

Example: You write a test suite to test controller functionality in an MVC application. The controller requires application data from a database as input during the test. There are two options to do this:

  • You hardcode which row(s) from the test database are selected as input (e.g. the 10th and 412th row)
  • You use a random number generator to pseudorandomly pick the data from the database (two rows selected by a random number generator)

The first is deterministic: every run of the test for the same revision of code should yield the same result. The second is non-deterministic: every run of the test suite has the possibility to yield a different result. The randomly picked data might however be a better representation of data edge cases. It might simulate a user feeding our controllers with unpredictable data better?

What are reasons to choose one over the other?

7
  • 6
    That test just fails, sometimes. martinfowler.com/articles/nonDeterminism.html
    – user40980
    Dec 17, 2013 at 13:54
  • 2
    Thanks for that link. With that article in mind, I felt I needed to clarify that non-determinism means in the context of this test suite. Because data is selected randomly from a database, all data fed to the controller is valid data by default. This means that false negatives don't exist in the test suite when it comes to the non determinism. In a way, this randomness simulates a user selecting data 'at random' for use in a controller. This is not necessarily the same non-determinism the article discusses, right?
    – DCKing
    Dec 17, 2013 at 14:15
  • 3
    recommended reading: Why is asking a question on “best practice” a bad thing?
    – gnat
    Dec 17, 2013 at 14:19
  • 13
    @DCKing: Consider what happens if your test fails. Okay, you have a bug. Uh, now what? Run it again in debug mode! Where it succeeds! Like it does the next hundred times you run it, and then you write off the issue as a cosmic ray strike. Non-determinisim in tests sounds absolutely unworkable. If you feel the need to cover more ground in your test cases, cover more ground. Initilise your RNG with a set seed and run the "test" a few hundred times with consistently random values.
    – Phoshi
    Dec 17, 2013 at 14:35
  • 2
    (finally got around to a machine where I could properly search twitter - the "That test just fails sometimes" is from the #FiveWordTechHorrors on Twitter - wanted to properly credit it)
    – user40980
    Dec 17, 2013 at 15:25

6 Answers 6

41

When every run of the test suite gives you the possibility to yield a different result, the test is almost completely worthless - when the suite shows you a bug, you have a high chance that you cannot reproduce it, and when you try to fix the bug, you cannot verify whether your fix works (or not).

So when you think you need to use some kind of random number generator for generating of your test data, either make sure you always initialize the generator with the same seed, or persist your random test data in a file before feeding it into your test, so you can re-run the test again with exact the same data from the run before. This way, you can transform any non-deterministic test into a deterministic one.

EDIT: Using a random number generator to pick some test data is IMHO sometimes a sign for being too lazy about picking good test data. Instead of throwing 100,000 randomly choosen test values and hope that this will be enough to discover all serious bugs by chance, better use your brain, pick 10 to 20 "interesting" cases, and use them for the test suite. This will not only result in a better quality of your tests, but also in a much higher performance of the suite.

8
  • Thanks for your answer. What is your opinion on the comment I made to my question?
    – DCKing
    Dec 17, 2013 at 14:18
  • 1
    @DCKing: if your really think a random generator will be better in picking good test cases than you (what I doubt), use it once to find combinations of test data where your program fails, and put those combinations into the "hardcoded" part of your test suite.
    – Doc Brown
    Dec 17, 2013 at 14:39
  • Thanks again. Updated my answer so that it doesn't seem to apply to just MVC apps.
    – DCKing
    Dec 17, 2013 at 16:07
  • 1
    In some UI contexts (for instance, games taking controller input) having test programs that generate random key entry can be useful for stress testing. They can uncover defects that are hard to find with deliberate input. Dec 17, 2013 at 18:56
  • 1
    What if when your random test fails, it outputs the fail scenario, which you can then use to further troubleshoot the problem? This is the case with property-based testing in frameworks like QuickCheck (Haskell) and ScalaCheck (Scala).
    – Andres F.
    Dec 12, 2016 at 14:30
12

Both.

Deterministic and nondeterministic tests have different use cases and different values to your suite. Generally nondeterministic can't provide the same precision as deterministic testing, which has slowly grown into "nondeterministic testing provides no value." This is false. They may be less precise, but they can also be much broader, which has its own benefits.

Let's take an example: you write a function that sorts a list of integers. What would be some of the deterministic unit tests you'd find useful?

  • An empty list
  • A list with just one element
  • A list with all of the same element
  • A list with multiple unique elements
  • A list with multiple elements, some of which are duplicates
  • A list with NaN, INT_MIN, and INT_MAX
  • A list that's already partially sorted
  • A list with 10,000,000 elements

And that's just a sorting function! Sure, you could argue that some of these are unnecessary, or that some of these can be ruled out with informal reasoning. But we're engineers and we've seen informal reasoning blow up in our face. We know we're not smart enough to completely understand the systems we've built or fully keep the complexity in our heads. That's why we write tests in the first place. Adding nondeterministic testing just says that we might not necessarily be smart enough to know all of the good tests a priori. By throwing semi-random data into your function, you're much more likely to find an edge case you missed.

Of course, that doesn't rule out deterministic testing either. Nondeterministic testing helps find bugs in huge swaths of the program. Once you've found the bugs, though, you need a reproducible way to show that you fixed it. So:

  • Use nondeterministic tests to find bugs in your code.
  • Use deterministic tests to verify fixes in your code.

Note that this means a lot of solid advice about unit tests don't necessarily apply to nondeterministic tests. For example, that they must be fast. Low-level property tests should be fast, but a nondeterministic test like "simulate a user randomly clicking buttons on your website and make sure you never get a 500 error" should favor comprehensiveness over speed. Just have a test like that run independently of your build process so that it's not slowing down development. For example, run it on its own private staging box.

6
  • Actually, I don't understand why non-deterministic tests should have any benefits over tests with pseudo-random, but deterministic test data.
    – Doc Brown
    Jun 24, 2021 at 13:40
  • 1
    @DocBrown are you familiar with fuzzing?
    – Hovercouch
    Jun 25, 2021 at 4:19
  • @DocBrown Well if you run it once you know that case works. Running fifty different test runs is more likely to find a new problem, than running the same test fifty times.
    – user253751
    Jun 29, 2021 at 8:50
  • @user253751: still one has to make sure in case a problem occurs, one can fix it and run the same test again which failed before. So there must be some precaution taken to make things deterministic again.
    – Doc Brown
    Jun 29, 2021 at 9:36
  • @DocBrown Most generative testing tools provide the run seed and inputs that failed, so you can either rerun it or extract it to an example test. More sophisticated tools, like Hypothesis, also keep a failed input database for later auditing and reuse.
    – Hovercouch
    Jun 29, 2021 at 17:28
5

Both deterministic and non-deterministic have a place

I would divide them as follows:

Unit tests.

These should have deterministic, repeatable tests with the the exact same data every time. Unit tests accompany specific, isolated code sections and should test them in a deterministic fashion.

Functional and input stress tests.

These can use the non-deterministic approach with the following caveats:

  • that fact is clearly delineated and called out
  • the random values selected are logged and can be re-tried manually
0

Random unit tests are good, but can miss corner or edge cases, which should be tested deterministically. Generally, random unit tests should have an invariant.

I've written a set of tests for a matrix arithmetic module. One test makes a matrix with all ones on the diagonal, random numbers on one side, and zeros on the other. The determinant of this is guaranteed to be 1; that's the invariant. Then it permutes the rows and columns with an even permutation; the determinant is still 1. Then it calculates the determinant by calling the module. The row operations multiply rows by random numbers, introducing roundoff error, but the result is still close to 1.

Another test calls a function which returns the intersection type of two line segments. The edge case of three of the endpoints being collinear is unlikely to result from random choice, so I test it deterministically.

0

My two cents.

I could be testing that the users can do things in my app, let's say that one of those things/test requires that my user is >18 years old.

In this case my test could look something like:

public void User_Can_Access_Content_When_18()
{
  string userName = GetRandom.String();
  string userEmail = GetRandom.String();
  int year = GetRandom.Int();
  int month = GetRandom.Int();
  int day = GetRandom.Int();
  string dob = $"{day}/{month}/{year}";

  bool result = UserCanAccessContentWhen18(dob, userName, userEmail);
  
  Assert.IsTrue(result);
}

In this case is obvious that my test will fail randomly, if I want to Assert that the user can access adult content. Instead I would write:

public void User_Can_Access_Content_When_18()
{
  string userName = GetRandom.String();
  string userEmail = GetRandom.String();

  DateTime today = DateTime.Now;
  int year = GetRandom.Int(today.Year - 18, today.Year - 150);
  int month = today.Month;
  int day = today.Day-1;
  string dob = $"{day}/{month}/{year}";

  bool result = UserCanAccessContentWhen18(dob, userName, userEmail);
  
  Assert.IsTrue(result);
}

This second test is a mix of fix and random values, that is how they normally are on my unit tests, there has to be some constraints for the values we want to test. (This test only tests for a random dob year, we would like to have more tests to cover other scenarios, but that is not the point here)

Now my test can still fail for different reasons:

  1. We could have logic that reads the dob string parameter and manipulates the string in order to get a simple number let's call it yearsOld, but this transformations only take a two digits number(why? well we do strange things sometimes), so when the person is >99 the test fails
  2. Maybe we transform the dob string parameter into a DateTime and then run some date comparison and well sometimes fails when it shouldn't... (I saw something like that happening with a phone number regex filter...)
  3. It could fail because the username has some special character that makes the code break

Following up on those 3 cases:

  1. Is it bad that the test fails sometimes? Yes of course, would it be better to hardcode a value for the year, let's say today.year - 18, or as some people suggests a few: -18, -42, -99 and not know that the test fails until a customer calls and says I can't access my content? I don't think that is better. Some would say I should have known better and add the number 100 on the list(of course I could but maybe I should know even better and not have made the mistake on the code on the first place...if I was that good maybe I would not need unit tests on the first place...)
  2. Same as 1, but it will fail less often. So chances are it will fail after two months the code was merge. In this case How am I supposed to know that the logic fails for some very specific cases?
  3. Same as 2, but it fails for something else, not even related to the parameter being tested, dob.

This examples will be painful to solve/reproduce unless we use some of the tools people have mention in previous answers, tools that allows you to repeat the test or to look at what data was randomly generated. But, Would it be better to have hardcoded values and not know that the code has bugs until the customer finds them out? I don't think so, this should be the developers pain, not the customer.

I've been using random generated values for two years now and I haven't saw random failures until a few days ago. I don't use those tools that provide info about the values used during the test so I've to manually find out myself. Was it painful? well a bit shocking at first, then I did what we do, I already knew which unit test was failing, so that narrow things a bit, then I knew some of the variables where randomly generated, so next step was to add logging info to know what values made my test fail, in my case it didn't took long to find out but next time it should take a bit less since I will mechanically add extra logs and have the code run until it fails(that is a function of ReSharper, or I could create an infinite loop ...)

I understand that randomly failing test make some developers question unit tests reliability but I'm not sure that is the right actitude. Knowing the weakness of my code before my customers find out beats it for me

Edit On top of what I said above, one of the reasons I like to use random generators is that it makes my unit tests more readable and therefore easier to maintain. Below a hardcoded unit test vs one with random generated values

public void RegisterMovement_Returns_False_When_DestinationLongitude_Is_Wrong()
{
  string satelliteName = "test satellite name";
  string satelliteId = "test satellite id";
  int userId = 5;
  int userRole = (int)Privilege.GroupAdministrator;
  double currentLatitude = 30.0;
  double currentLongitude = 30.0;
  double destinationLatitude = 30.0;
  double destinationLongitude = 181.0;

  bool result = RegisterMovement(satelliteId, satelliteName, userId, userRole, currentLatitude, currentLongitude, destinationLatitude, destinationLongitude);

  Assert.IsFalse(result);
}
public void RegisterMovement_Returns_False_When_DestinationLongitude_Is_Wrong()
{
  string satelliteName = GetRandom.String();
  string satelliteId = GetRandom.String();
  int userId = GetRandom.Int();
  int userRole = GetRandom.UserRole();
  double currentLatitude = GetRandom.Latitude();
  double currentLongitude = GetRandom.Longitude();
  double destinationLatitude = GetRandom.Latitude();
  double destinationLongitude = GetRandom.Double(181, double.MaxValue);
            
  bool result = RegisterMovement(satelliteId, satelliteName, userId, userRole, currentLatitude, currentLongitude, destinationLatitude, destinationLongitude);

  Assert.IsFalse(result);
}

When I see the word GetRandom, and nothing specific inside, then I automatically know that the value is not relevant for this test, I will focus on the other/s values

-2

You don't really want deterministic vs. non-deterministic.

What you might want is "always the same" vs. "not always the same".

For example, you might have a build number which increases with each build, and when you want some random numbers, you initialise a random number generator with the build number as seed. So every build, you do your tests with different values, giving you more chances to find bugs.

But once a bug is found, all you need to do is run the test with the same build number, and it is reproducible.

1
  • 2
    Or if you don't have a build number to use, place the initial value of the seed in the output of the test run, so you can again re-run tests with the same seed. Dec 12, 2016 at 21:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.