Integration test ordering vs. before/after style setup/cleanup when one test is a subset of another

Question

I'm aware that it's generally considered bad style to write tests that have to run in a particular order. Let's say I'm testing an account-management API, and have the following two integration tests: test_create_account, and test_log_in. What I understand to be the correct style for this would be something to the effect of:

test_create_account:
  // check whether account creation works

before_test_log_in:
  // create a throwaway account
test_log_in:
  // try logging into the throwaway account from the 'before_' fixture

However, if I can't control the order in which they run (or set up some sort of inter-test dependencies to enforce at least partial ordering), I'll get failures in both tests and have to paw through to figure out which test case is most relevant to the fact that I have failing tests. If I could order/dependency-ify them, it would make the output from the test runner much less noisy.

Why (if at all) is it more important for the tests to be fully independent than to have more eloquent output?

Answer 1

Why should tests be independent from each other?

Because otherwise you cannot run them individually. This becomes a challenge when one of a few hundred test fails in a full run of your test suit - but when you are trying to find the cause, running the formerly failing test alone does not fail any more. More general, assume you pick a certain subset of the tests and always get different results depending on what you picked - good luck in managing this.

To prevent such side effects, each of the tests should start with a fixed, defined state of the system under test (this holds regardless if you are doing unit or integration tests). For example, when you have an integration test which includes a database, before the test is run, it requires a fresh copy of the database, maybe with a certain preset of accounts or test data tailored for the specific test. There should be a "setup" state at the beginning of each test which provides this database and its content, regardless of the operations which took place beforehand.

test_create_account may, for example, require a clean test database with no account so far. test_log_in may require, for example, a clean test database with a predefined account. If you want to make it possible to reuse the same test database for both tests, to save some space, there is nothing wrong when test_log_in uses the same create_account method on the db like test_create_account to create the required throwaway account.

Of course, it depends heavily of the system (for example, the DMBS), how costly it is to setup a fresh new copy of the DB for each test. For a lightweight database like SQLite this maybe pretty simple - just copy a prepared DB file from a fixed origin. For a DBMS like Oracle, you may find it simpler and quicker to have only one DB instance for testing and run some general cleanup scripts beforehand. Other systems may require completely different lines of action, but I guess you get the idea.

What about making the output from the test runner much less noisy?

It is fully up to you what "noise" your tests produce. Ideally, when a test does not fail, it should produce almost no output, maybe just a message "test XYZ succeeded" (or maybe not even this, when you have a statistics, telling you something like "2 of 2 tests succeeded"). So there should be no compelling reason why two independent, succeeding tests produce more than two lines of output, regardless of the order in which you run them.

If both tests reuse some methods (like create_account), don't implement any directly visible output in create_account. Make that method throw an exception in case it fails, and let the tests catch the exception and produce failure messages from it. create_account may write something into some separate background log file, but I would not show the content of the log in each test run, only bring it to the front when a test fails.

Answer 2

before_test_log_in requires an account as a precondition, but that doesn't mean it needs to create an account in the same "realistic" way as is tested in test_create_account.

1. test_create_account should take input data (username, email, password, etc.) from a factory or fixture, and spit out an account object that can be asserted against (make sure that it has the correct username, email, password, etc.)

2. before_test_log_in should take an account, and assert that it can log in (whatever that entails). This doesn't have to duplicate the logic of account creation as tested in test_create_account. You can just directly make an account object (using a factory or fixture).