18

I'm trying to get to grips with unit testing.

Say we have a die which can has a default number of sides equal to 6 (but can be 4, 5 sided etc.):

import random
class Die():
    def __init__(self, sides=6):
        self._sides = sides

    def roll(self):
        return random.randint(1, self._sides)

Would the following be valid/useful unit tests?

  • test a roll in the range 1-6 for a 6 sided die
  • test a roll of 0 for a 6 sided die
  • test a roll of 7 for a 6 sided die
  • test a roll in the range 1-3 for a 3 sided die
  • test a roll of 0 for a 3 sided die
  • test a roll of 4 for a 3 sided die

I'm just thinking that these are a waste of time as the random module has been around for long enough but then i think if the random module gets updated (say i update my Python version) then at least i'm covered.

Also, do I even need to test other variations of die rolls e.g. the 3 in this case, or is it good to cover another initialized die state?

  • 1
    What about a minus-5-sided die, or a null-sided die? – JensG Sep 11 '14 at 14:30
22

You are right, your tests should not verify that the random module is doing its job; a unittest should only test the class itself, not how it interacts with other code (which should be tested separately).

It is of course entirely possible that your code uses random.randint() wrong; or you be calling random.randrange(1, self._sides) instead and your die never throws the highest value, but that'd be a different kind of bug, not one you could catch with a unittest. In that case, your die unit is working as designed, but the design itself was flawed.

In this case, I'd use mocking to replace the randint() function, and only verify that it has been called correctly. Python 3.3 and up comes with the unittest.mock module to handle this type of testing, but you can install the external mock package on older versions to get the exact same functionality

import unittest
try:
    from unittest.mock import patch
except ImportError:
    # < python 3.3
    from mock import patch


@patch('random.randint', return_value=3)
class TestDice(unittest.TestCase):
    def _make_one(self, *args, **kw):
        from die import Die
        return Die(*args, **kw)

    def test_standard_size(self, mocked_randint):
        die = self._make_one()
        result = die.roll()

        mocked_randint.assert_called_with(1, 6)
        self.assertEqual(result, 3)

    def test_custom_size(self, mocked_randint):
        die = self._make_one(sides=42)
        result = die.roll()

        mocked_randint.assert_called_with(1, 42)
        self.assertEqual(result, 3)


if __name__ == '__main__':
    unittest.main()

With mocking, your test is now very simple; there are only 2 cases, really. The default case for a 6-sided die, and the custom sides case.

There are other ways to temporarily replace the randint() function in the global namespace of Die, but the mock module makes this easiest. The @mock.patch decorator here applies to all test methods in the test case; each test method is passed an extra argument, the mocked random.randint() function, so we can test against the mock to see if it it indeed has been called correctly. The return_value argument specifies what is returned from the mock when it is called, so we can verify that the die.roll() method indeed returned the 'random' result to us.

I've used another Python unittesting best practice here: import the class under test as part of the test. The _make_one method does the importing and instantiation work within a test, so that the test module will still load even if you made a syntax error or other mistake that'll prevent the original module to import.

This way, if you made a mistake in the module code itself, the tests will still be run; they'll just fail, telling you about the error in your code.

To be clear, the above tests are simplistic in the extreme. The goal here is not to test that random.randint() has been called with the right arguments, for example. Instead, the goal is to test that the unit is producing the right results given certain inputs, where those inputs include the results of other units not under test. By mocking the random.randint() method you get to take control over just another input to your code.

In real world tests, the actual code in your unit-under-test is going to be more complex; the relationship with inputs passed to the API and how other units are then invoked can be interesting still, and mocking will give you access to intermediate results, as well as let you set the return values for those calls.

For example, in code that authenticates users against a 3rd party OAuth2 service (a multi-stage interaction), you want to test that your code is passing the right data to that 3rd party service, and lets you mock out different error responses that that 3rd party service would return, letting you simulate different scenarios without having to build a full OAuth2 server yourself. Here it is important to test that information from a first response have been handled correctly and have been passed on to a second stage call, so you do want to see that the mocked service is being called correctly.

  • 1
    You've got quite a few more than 2 test cases... results check for default value: lower (1), upper (6), below lower (0), beyond upper (7) and results for user specified numbers like max_int etc. input also isn't validated, which may need to be tested for at some point... – James Snell Jan 26 '14 at 11:06
  • 2
    No, those are tests for randint(), not the code in Die.roll(). – Martijn Pieters Jan 26 '14 at 11:09
  • There's actually a way to ensure that not just randint is called correctly but that its result is used correctly too: mock it to return a sentinel.die for example (sentinel object is from unittest.mock too) and then verify that it is what was returned from your roll method. This actually allows only one way of implementing the tested method. – aragaer Mar 24 '14 at 17:09
  • @aragaer: sure, if you want to verify that the value is returned unchanged, sentinel.die would be a great way to ensure that. – Martijn Pieters Mar 24 '14 at 17:11
  • I don't understand why you would want to ensure that mocked_randint is called_with certain values. I understand wanting to mock randint to return predictable values, but isn't the concern just that it returns predictable values and not what values it is called with? It seems to me that checking the called values is unnecessarily tying the test to fine details of implementation. Also why do we care that the die returns the exact value of randint? Don't we really just care that it returns a value > 1 and less than equal to the max? – bdrx Sep 11 '14 at 14:14
16

Martijn's answer is how you'd do it if you really wanted to run a test that demonstrates that you're calling random.randint. However, at the risk of being told "that doesn't answer the question", I feel this shouldn't be unit tested at all. Mocking randint is no longer black box testing - you're specifically showing that certain things are going on in the implementation. Black box testing it isn't even an option - there is no test you can execute that will prove that the result will never be less than 1 or more than 6.

Can you mock randint? Yes, you can. But what are you proving? That you called it with arguments 1 and sides. What does that mean? You're back in square one - at the end of the day you'll end up having to prove - formally or informally - that calling random.randint(1, sides) correctly implements a dice roll.

I'm all for unit testing. They're fantastic sanity checks and expose the presence of bugs. However, they can never prove their absence, and there's things that can't be asserted through testing at all (e.g. that a particular function never throws an exception or always terminates.) In this particular case, I feel there's very little you stand to gain. For behavior that's deterministic, unit tests make sense because you actually know what the answer you're expecting will be.

  • Unit tests are not black box tests, really. That's what integration tests are for, to see to it that the various parts interact as designed. It's a matter of opinion, of course (most testing philosophy is), see Does "Unit Testing" falls under white box or black box testing? and Black Box Unit Testing for some (Stack Overflow) perspectives. – Martijn Pieters Mar 24 '14 at 13:09
  • @MartijnPieters I disagree that "that's what integration tests are for". Integration tests are for checking that all the components of the system interact correctly. They're not the place to test that a given component gives the correct output for a given input. As for black box vs white box unit testing, white box unit tests will eventually break with implementation changes, and any assumptions you've made in the implementation will likely carry over into the test. Validating that random.randint is called with 1, sides is worthless if that's the wrong thing to do. – Doval Mar 24 '14 at 13:25
  • Yes, that's a limitation of a white-box unit test. However, there is no point in testing that random.randint() will correctly return values in the range [1, sides] (inclusive), that's up to the Python developers to ensure that the random unit works correctly. – Martijn Pieters Mar 24 '14 at 13:31
  • And as you say yourself, unit testing cannot guarantee that your code is bug free; if your code is using other units wrongly (say, you expected random.randint() to behave like random.randrange() and thus call it with random.randint(1, sides + 1), then you are sunk anyway. – Martijn Pieters Mar 24 '14 at 13:33
  • 2
    @MartijnPieters I agree with you there, but that's not what I'm objecting to. I'm objecting to testing that random.randint is being called with arguments (1, sides). You have assumed in the implementation that this is the correct thing to do, and now you are repeating that assumption in the test. Should that assumption be wrong, the test will have passed but your implementation is still incorrect. It's a half-assed proof that's a full pain-in-the-ass to write and maintain. – Doval Mar 24 '14 at 13:40
6

Fix random seed. For 1, 2, 5 and 12-sided dice, confirm that a few thousand rolls gives results including 1 and N, and not including 0 or N + 1. If by seem freak chance you get a set of random results that don't cover the expected range, switch to a different seed.

Mocking tools are cool, but just because they allow you to do a thing doesn't mean that thing should be done. YAGNI applies to test fixtures as much as features.

If you can easily test with unmocked dependencies, you pretty much always should; that way your tests will be focused on reducing defect counts, not just increasing test count. Excess mocking risks creating misleading coverage figures, which in turn can lead to postponing the actual testing to some later phase you perhaps never have time to get round to...

3

What is a Die if you think about it ? - no more than a wrapper around random. It encapsulates random.randint and relabels it in terms of your application's own vocabulary : Die.Roll.

I don't find it relevant to insert another layer of abstraction between Die and random because Die itself is already this layer of indirection between your application and the platform.

If you want canned dice results, just mock Die, don't mock random.

In general, I don't unit test my wrapper objects that communicate with external systems, I write integration tests for them. You could write a couple of those for Die but as you pointed out, due to the random nature of the underlying object, they will not be meaningful. In addition, there's no configuration or network communication involved here so not much to test except a platform call.

=> Considering that Die is only a few trivial lines of code and adds little to no logic compared to random itself, I'd skip testing it in that specific example.

2

Seeding the random number generator and verifying expected results is NOT, as far as I can see, a valid test. It makes assumptions as to HOW your dice works internally, which is naughty-naughty. The developers of python could change the random number generator, or the die (NOTE: "dice" is plural, "die" is singular. Unless your class implements multiple die rolls in one call, it should probably be called "die") could use a different random number generator.

Similarly, mocking the random function assumes that the class implementation works exactly as expected. Why might this not be the case? Someone might take control of the default python random number generator, and to avoid that, a future version of your die may fetch several random numbers, or larger random numbers, to mix in more random data. A similar scheme was used by the makers of the FreeBSD operating system, when they suspected the NSA was tampering with the hardware random number generators built into CPUs.

If it were me, I'd run, say, 6000 rolls, tally them, and make sure that each number from 1-6 is rolled between 500 and 1500 times. I would also check that no numbers outside that range are returned. I might also check that, for a second set of 6000 rolls, when ordering the [1..6] in order of frequency, the result is different (this will fail once out of 720 runs, if the numbers are random!). If you want to be thorough, you might find the frequency of numbers following a 1, following a 2, etc; but make sure your sample size is big enough, and you have enough variance. Humans expect random numbers to have fewer patterns than they actually do.

Repeat for a 12 sided, and 2 sided die (6 is the most used, so is the most expected for anyone writing this code).

Finally, I would test to see what happens with a 1-sided die, a 0 sided die, a -1 sided die, a 2.3 sided die, a [1,2,3,4,5,6] sided die, and a "blah"-sided die. Of course, these should all fail; do they fail in a useful way? These should probably fail on creation, not on rolling.

Or, perhaps, you want too handle these differently - perhaps creating a die with [1,2,3,4,5,6] should be acceptable - and perhaps "blah" as well; this might be a die with 4 faces, and each face having a letter on it. The game "Boggle" springs to mind, as does a magic eight ball.

And finally, you might want to contemplate this: http://lh6.ggpht.com/-fAGXwbJbYRM/UJA_31ACOLI/AAAAAAAAAPg/2FxOWzo96KE/s1600-h/random%25255B3%25255D.jpg

2

At the risk of swimming against the tide, I solved this exact problem a number of years ago using a method not so far mentioned.

My strategy was simply to mock the RNG with one that produces a predictable stream of values spanning the entire space. If (say) side=6 and the RNG produces values from 0 to 5 in sequence I can predict how my class should behave and unit test accordingly.

The rationale is that this tests the logic in this class alone, on the assumption that the RNG will eventually produce each of those values and without testing the RNG itself.

It's simple, deterministic, reproducible and it does catch bugs. I would use the same strategy again.


The question does not spell out what the tests should be, just what data might be used for testing, given the presence of an RNG. My suggestion is merely to test exhaustively by mocking the RNG. The question of what is worth testing depends on information not provided in the question.

  • Say you mock the RNG to be predictable. Well what do you then test? The question asks "Would the following be valid/useful unit tests?" Mocking it to return 0-5 is not a test but rather test setup. How would you "unit test accordingly"? I am failing to understand how it "does catch bugs". I am having a hard time understanding what I need to 'unit' test. – bdrx Sep 11 '14 at 13:52
  • @bdrx: This was a while ago: I would answer it differently now. But see edit. – david.pfx Sep 11 '14 at 14:15
1

The tests you suggest in your question don't detect a modular arithmetic counter as implementation. And they don't detect common implementation errors in probability distribution related code like return 1 + (random.randint(1,maxint) % sides). Or a change to the generator that results in 2-dimensional patterns.

If you actually want to verify that you are generating evenly distributed random-appearing numbers you need to check a very wide variety of properties. To do a reasonably good job at that you could run http://www.phy.duke.edu/~rgb/General/dieharder.php on your generated numbers. Or write a similarly complex unit-test suite.

That's not the fault of unit-testing or TDD, randomness just happens to be a very difficult property to verify. And a popular topic for examples.

-1

The easiest test of a die-roll is simply to repeat it several hundred-thousand times, and validate that each possible result was hit roughly (1 / number of sides) times. In the case of a 6-sided die, you should see each possible value hit about 16.6% of the time. If any are off by more than a percent, then you have a problem.

Doing it this way avoids allows you to refactor the underlying mechanic of generating a random number easily, and most importantly, without changing the test.

  • 1
    this test would pass for a totally non-random implementation that simply loops through sides one by one in a predefined order – gnat Mar 24 '14 at 19:47
  • 1
    If a coder is intent on implementing something in bad faith (not using a randomizing agent on a die), and simply trying to find something to 'make the red lights turn green' you have more problems than unit testing can really solve. – ChristopherBrown Mar 24 '14 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.