Is there a better way to run sequential tests?

Question

I'm using unittest framework to run Python tests. Some system tests consist of a series of steps for a given scenario, and they need to run in a specific order.

For example, I have a client application which performs offsite backups. System tests run on a live API in order to check high-level regressions in the code of the client which made it somehow incompatible with the API. The steps look like that:

1. Perform the original off-site backup of a directory.
2. Add a file to the directory. Perform a new backup, but simulate a situation where the API crashed in the middle.
3. Launch backup again. Ensure the upload of the file from the previous step is resumed.
4. Move a file locally. Perform a backup. Check that everything works as expected.
5. Delete a file locally. Perform a backup, and while doing so, crash intentionally to simulate a situation where the file was deleted remotely, but the local client doesn't know that.
6. Perform a backup again. The client will ask the API to delete a file which was already deleted, receive a HTTP 404, and is expected to handle it correctly.
7. Recover a specific file.
8. Recover the whole directory.

Here, the order is crucial: I can't test the recovery step, if I haven't made the backup in the first place. And I can't test the third step before performing the second one, because I cannot resume an upload which never started in the first place.

In order to determine the order, I prefix the names of the tests with a number:

def test_07_recover_file(self):
   ...

def test_08_recover_everything(self):
   ...

This creates a problem when I need to insert a test in the middle of the chain. Instead of just adding the test, I need to add it first, and then go and change the names of all successive tests.

I thought about using for each test a decorator with the names of the tests which should run before it—a bit like dependency management, where every package can tell that it needs some other packages to be installed. I'm afraid, however, that such approach would be difficult to understand (and require a custom test runner).

Is there a more elegant approach to that?

Answer 1

Two possible options.

• sortTestMethodsUsing.

import unittest

class Test(unittest.TestCase):
    def test_07_recover_file(self):
       ...

    def test_08_recover_everything(self):
       ...

if __name__ == "__main__":
    test_order = ["test_07_recover_file", "test_08_recover_everything"] 
    test_loader = unittest.TestLoader()
    test_loader.sortTestMethodsUsing = lambda x, y: test_order.index(x) - test_order.index(y)
    unittest.main(testLoader=test_loader)

• TestSuite

import unittest

class WidgetTestCase(unittest.TestCase):
    def test_07_recover_file(self):
       ...

    def test_08_recover_everything(self):
       ...

def suite():
    suite = unittest.TestSuite()
    suite.addTest(WidgetTestCase('test_07_recover_file'))
    suite.addTest(WidgetTestCase('test_08_recover_everything'))
    return suite

if __name__ == '__main__':
    runner = unittest.TextTestRunner()
    runner.run(suite())

Answer 2

I think this is an example of the XY problem. You are looking for an answer to the question 'Is there a better way to run sequential tests?' but you're using a framework that is built around running tests in whatever order they're discovered (i.e. the name).

It seems fundamental to your task in hand that:

1. steps are performed sequentially, iff the previous step succeeds
2. steps must be done in the correct order
3. its' a non-requirement to jump into the sequence at some arbitrary position.

Therefore I think you need one test that does each of the sequential steps, but you could factor out each step to a separate method for ease of writing, particularly as some steps are very similar.

Answer 3

You don't. You shouldn't have any dependencies between unit tests, each test cases should be runnable independently. You can put steps that are common for all tests in the setUp() to restore the test environment to a well known state before each test case runs.

If you have complex tests that need multiple sequential steps, then break the test down into multiple helper functions and call those functions from the test function:

class MyTestCase(unittest.TestCase):
    def test_something_complex(self):
        backup_file = self.create_backup()
        self.add_file(...)

        with mock.patch(... simulate a crash ...):
            self.create_backup()
        self.assert_things_are_where_they_should_be()

        self.create_backup()
        self.assert_things_are_doing_well()

        self.move_file(..., ...)
        self.create_backup()
        self.assert_things_looks_good()

        ... and so on ...

    def test_another_complex_stuff(self):
        ...

You likely want to also break this long test into multiple independent tests and have a common setUp() function rather than one very long test. Having smaller independent tests makes it much easier to diagnose issues if the test case fails.