How to structure tests where one test is another test's setup?

Question

I'm integration testing a system, by using only the public APIs. I have a test that looks something like this:

def testAllTheThings():
  email = create_random_email()
  password = create_random_password()

  ok = account_signup(email, password)
  assert ok
  url = wait_for_confirmation_email()
  assert url
  ok = account_verify(url)
  assert ok

  token = get_auth_token(email, password)
  a = do_A(token)
  assert a
  b = do_B(token, a)
  assert b
  c = do_C(token, b)

  # ...and so on...

Basically, I'm attempting to test the entire "flow" of a single transaction. Each step in the flow depends on the previous step succeeding. Because I'm restricting myself to the external API, I can't just go poking values into the database.

So, either I have one really long test method that does `A; assert; B; assert; C; assert...", or I break it up into separate test methods, where each test method needs the results of the previous test before it can do its thing:

def testAccountSignup():
  # etc.
  return email, password

def testAuthToken():
  email, password = testAccountSignup()
  token = get_auth_token(email, password)
  assert token
  return token

def testA():
  token = testAuthToken()
  a = do_A(token)
  # etc.

I think this smells. Is there a better way to write these tests?

Answer 1

If this test is intended to run frequently, your concerns would rather be focused on how to present test results in a way convenient to those expected to work with these results.

From this perspective, testAllTheThings raises a huge red flag. Imagine someone running this test every hour or even more frequently (against buggy codebase of course, otherwise there would be no point to re-run), and seeing every time all the same FAIL, without a clear indication of what stage failed.

Separate methods look much more appealing, because results of re-runs (assuming steady progress in fixing bugs in code) could look like:

    FAIL FAIL FAIL FAIL
    PASS FAIL FAIL FAIL -- 1st stage fixed
    PASS FAIL FAIL FAIL
    PASS PASS FAIL FAIL -- 2nd stage fixed
    ....
    PASS PASS PASS PASS -- we're done

---

Side note, in one of my past projects, there were so many re-runs of dependent tests that users even began complaining about not willing to see repeated expected failures at later stage "triggered" by a failure at the earlier one. They said this garbage makes it harder to them to analyze test results "we know already that the rest will fail by test design, don't bother us repeating".

As a result, test developers were eventually forced to extend their framework with additional SKIP status and add a feature in test manager code to abort execution of dependent tests and an option to drop SKIPped test results from the report, so that it looked like:

    FAIL -- the rest is skipped
    PASS FAIL -- 1st stage fixed, abort after 2nd test
    PASS FAIL
    PASS PASS FAIL -- 2nd stage fixed, abort after 3rd test
    ....
    PASS PASS PASS PASS -- we're done

Answer 2

I'd separate the testing code from the setup code. Perhaps:

# Setup
def accountSignup():
    email = create_random_email()
    password = create_random_password()

    ok = account_signup(email, password)
    url = wait_for_confirmation_email()
    verified = account_verify(url)
    return email, password, ok, url, verified

def authToken():
    email, password = accountSignup()[:2]
    token = get_auth_token(email, password)
    return token

def getA():
    token = authToken()
    a = do_A()
    return a

def getB():
    a = getA()
    b = do_B()
    return b

# Testing
def testAccountSignup():
    ok, url, verified = accountSignup()[2:]
    assert ok
    assert url
    assert verified

def testAuthToken():
    token = authToken()
    assert token

def testA():
    a = getA()
    assert a

def testB():
    b = getB()
    assert b

Remember, all random information that is generated must be included in the assertion in case it fails, otherwise your test may not be reproducible. I might even record the random seed used. Also any time a random case does fail, add that specific input as a hard-coded test to prevent regression.

Answer 3

Not much better, but you can at least separate setup code from asserting code. Write a separate method that tells the entire story step by step, and take a parameter controlling how many steps it should take. Then each test can say something like simulate 4 or simulate 10 and then assert whatever it tests.

Answer 4

Well, I might not get the Python syntax right here by "air coding", but I guess you get the idea: you can implement a general function like this:

def asserted_call(create_random_email,*args):
    result=create_random_email(*args)
    assert result
    return result

which will allow you to write your tests like this:

  asserted_call(account_signup, email, password)
  url = asserted_call(wait_for_confirmation_email)
  asserted_call(account_verify,url)
  token = asserted_call(get_auth_token,email, password)
  # ...

Of course, it is debatable if the loss in readability of this approach is worth to use it, but it reduces the boilerplate code a little bit.