I've got a simple authentication solution, where a user can log in with either email or username.

Since I want to show the little feature in the acceptance tests but don't want to pollute them, I thought I'll introduce some randomness. Do you think this is bad or good?

step "Fill in my credentials" do
    fill_in :user_login, with: [@user.email, @user.name].sample
    fill_in :user_password, with: @user.password

4 Answers 4


I would avoid randomness in your automated tests. You want all of your automated tests to be predictable. Trying random inputs is certainly not predictable. That way if one of your tests fail, you will be able to more quickly determine why.

Instead, spend some time thinking about cases that might deserve testing and test those explicitly. Thinking about your code like this is more effective than using random tests and hoping that they catch cases you didn't think of.

  • 2
    If add(x, y) = x + y and we test add() by asserting add(x, y) == add(y, x), then we can use random x and y just fine. Random inputs don't have to be unpredictable, but you do have to make sure and show (or log) the test case if it's random and something breaks. QuickCheck is a testing library that was written precisely to help with testing using random values. Commented Mar 30, 2013 at 2:14
  • 1
    If your test is only dealing with a small number of randomly generated values per test, recording every random value is sufficient to allow reproduction of the test results. If this is unwieldy saving the seed is often sufficient to allow the test to be reproduced. Of course, this sort of testing can become difficult if working with randomized algorithms, unless they accept a random number generator as a parameter. All that being said, randomness is inappropriate unless you are testing so many values that you have no choice.
    – Brian
    Commented Mar 30, 2013 at 15:36

Let's suppose that it works for email, but not name.

Now your acceptance test will fail half the time. Just re-running the test will potentially change the result. That'll make it harder to notice errors and track down the problem with it occurs. It'll be way easier if you just write two tests, one for each login method.


I won't state randomness is a bad idea.

[Sav07] tested his algorithm implementation for binary search by writing a group of test which included some "theories" (hypotheses) about how the search algorithm should perform.

He then ran the "thorough" group of tests with a lot of random fixtures.

In Ruby:

require "test/unit"
require "binary_search" # binary_search gem: 
                        # <https://github.com/tyler/binary_search>

class BinarySearchTestTheories < Test::Unit::TestCase
    def before_all
       @rand = Random.new()

    def test_theories
        experiments = 1000
        max_value = 1073741824 # Fixnum max value

        while (experiments-- > 0) do
            test_array = generateRandomSortedArray

            target_must_exist = [true, false].sample
            if target_must_exist
                target = test_array[@rand.rand(test_array.length)]
                target = @rand.integer(max_value)

            return_value = test_array.binary_search { |v| target <=> v }

            assert_theory1(test_array, target, return_value)
            assert_theory2(test_array, target, return_value)
            assert_theory3(test_array, target, return_value)
            assert_theory4(test_array, target, return_value)

    # Theories
    def assert_theory1(arr, target, expectation) ... end
    def assert_theory2(arr, target, expectation) ... end
    def assert_theory3(arr, target, expectation) ... end
    def assert_theory4(arr, target, expectation) ... end

    # Helper
    def does_array_contain_target?(arr, target) ... end
    def target_position(arr, target) ... end

I'm not certain his "test theories" qualify as acceptance tests---but they clearly are beyond unit or "boundary" tests.

My conclusion: random acceptance tests are useful for predictable algorithms. Sampling email and user name fixtures seems trivial enough.

[Sav07] Alberto Savoia: "Beautiful Tests", in: Andy Oram and Greg Wilson (eds.): Beautiful Code, Beijing: O'Reilly, 2007.

See the full article at: http://cdn.ttgtmedia.com/searchSoftwareQuality/downloads/BeautifulTests.pdf

  • Somehow the answers start to look like my question ;) There doesnt seem to be a general solution to this kind of answer. Commented Apr 30, 2013 at 11:03

I sometimes write tests that depend on random occurrences for whether they succeed or not. I've learned that there are a few caveats, as described by other answers:

  • You do not ideally want a test that only fails sometimes. If you break your code, you want to find out when you integrate your change, so you know which change to undo.
  • If there are only a small number of inputs that cause failure, you need to know which inputs they are in order to fix the problem.

The former problem can be mostly solved, as long as you are willing to accept a slow test suite, by running the test many times. The latter problem requires you to log any random factors that may cause the failure.

These techniques also apply to situations where the randomness is unavoidable, e.g. you are testing for potential timing-related conditions in a multithreaded program.

As long as you are careful, it can be done. But the best thing to do is to isolate potential failure conditions and write a predictable test that always fails for them prior to fixing the problem, IMHO.

  • 1
    Well, once you catch a failure via randomness, that failure should be turned into its own test, either with nonrandom parameters or a hard-coded seed. In the OP's case, may as well just start by doing that, so no randomness.
    – Brian
    Commented Mar 30, 2013 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.