6

I'm using unittest framework to run Python tests. Some system tests consist of a series of steps for a given scenario, and they need to run in a specific order.

For example, I have a client application which performs offsite backups. System tests run on a live API in order to check high-level regressions in the code of the client which made it somehow incompatible with the API. The steps look like that:

  1. Perform the original off-site backup of a directory.
  2. Add a file to the directory. Perform a new backup, but simulate a situation where the API crashed in the middle.
  3. Launch backup again. Ensure the upload of the file from the previous step is resumed.
  4. Move a file locally. Perform a backup. Check that everything works as expected.
  5. Delete a file locally. Perform a backup, and while doing so, crash intentionally to simulate a situation where the file was deleted remotely, but the local client doesn't know that.
  6. Perform a backup again. The client will ask the API to delete a file which was already deleted, receive a HTTP 404, and is expected to handle it correctly.
  7. Recover a specific file.
  8. Recover the whole directory.

Here, the order is crucial: I can't test the recovery step, if I haven't made the backup in the first place. And I can't test the third step before performing the second one, because I cannot resume an upload which never started in the first place.

In order to determine the order, I prefix the names of the tests with a number:

def test_07_recover_file(self):
   ...

def test_08_recover_everything(self):
   ...

This creates a problem when I need to insert a test in the middle of the chain. Instead of just adding the test, I need to add it first, and then go and change the names of all successive tests.

I thought about using for each test a decorator with the names of the tests which should run before it—a bit like dependency management, where every package can tell that it needs some other packages to be installed. I'm afraid, however, that such approach would be difficult to understand (and require a custom test runner).

Is there a more elegant approach to that?

1
  • 2
    I can't speak for Python, but I find Gherkin syntax inside .feature files to be an elegant way of orchestrating complex multi-step tests such as this using .NET with SpecFlow. Based on a quick search, it looks like 'behave' is one of the Python equivalents to SpecFlow (Support for the same Gherkin feature files, tied to test steps using decorators), although I couldn't really comment how well it might integrate with unittest; I'd hope that might be an already-solved problem though. – Ben Cottrell Jan 31 at 0:49
6

Two possible options.

import unittest

class Test(unittest.TestCase):
    def test_07_recover_file(self):
       ...

    def test_08_recover_everything(self):
       ...

if __name__ == "__main__":
    test_order = ["test_07_recover_file", "test_08_recover_everything"] 
    test_loader = unittest.TestLoader()
    test_loader.sortTestMethodsUsing = lambda x, y: test_order.index(x) - test_order.index(y)
    unittest.main(testLoader=test_loader)
import unittest

class WidgetTestCase(unittest.TestCase):
    def test_07_recover_file(self):
       ...

    def test_08_recover_everything(self):
       ...

def suite():
    suite = unittest.TestSuite()
    suite.addTest(WidgetTestCase('test_07_recover_file'))
    suite.addTest(WidgetTestCase('test_08_recover_everything'))
    return suite

if __name__ == '__main__':
    runner = unittest.TextTestRunner()
    runner.run(suite())
2

I think this is an example of the XY problem. You are looking for an answer to the question 'Is there a better way to run sequential tests?' but you're using a framework that is built around running tests in whatever order they're discovered (i.e. the name).

It seems fundamental to your task in hand that:

  1. steps are performed sequentially, iff the previous step succeeds
  2. steps must be done in the correct order
  3. its' a non-requirement to jump into the sequence at some arbitrary position.

Therefore I think you need one test that does each of the sequential steps, but you could factor out each step to a separate method for ease of writing, particularly as some steps are very similar.

1
  • This was one of the suggestions in Lie Ryan's answer below. While it can work, a single test, however, has a drawback of a less explicit granularity (not lower, but less explicit). What I mean is that with eight tests in a chain, if I see that test_05_delete_file fails, I immediately know that the problem is with the synchronization of the deleted files, and often this is enough to guess what I did wrong in my last commit. If I have only one test, I need to step in, and search the stack trace to find the exact location where the test failed. Not impossible, but still. – Arseni Mourzenko Feb 4 at 23:31
-1

You don't. You shouldn't have any dependencies between unit tests, each test cases should be runnable independently. You can put steps that are common for all tests in the setUp() to restore the test environment to a well known state before each test case runs.

If you have complex tests that need multiple sequential steps, then break the test down into multiple helper functions and call those functions from the test function:

class MyTestCase(unittest.TestCase):
    def test_something_complex(self):
        backup_file = self.create_backup()
        self.add_file(...)

        with mock.patch(... simulate a crash ...):
            self.create_backup()
        self.assert_things_are_where_they_should_be()

        self.create_backup()
        self.assert_things_are_doing_well()

        self.move_file(..., ...)
        self.create_backup()
        self.assert_things_looks_good()

        ... and so on ...

    def test_another_complex_stuff(self):
        ...

You likely want to also break this long test into multiple independent tests and have a common setUp() function rather than one very long test. Having smaller independent tests makes it much easier to diagnose issues if the test case fails.

5
  • 3
    -1. Your answer is perfectly valid in a context of unit tests. My question, however, has nothing to do with unit tests. – Arseni Mourzenko Jan 31 at 11:17
  • 2
    @ArseniMourzenko: I stand by my answer, my answer has nothing to specific about unit test; no matter whether it's unit tests or integration tests or system tests, a test should be independently runnable. Trying to do otherwise will make your tests much harder to maintain and run. Sequencing of subTest() can be done as just calling functions from a test to name subsections of a test and to have multiple dots per test in the report. – Lie Ryan Jan 31 at 22:35
  • 1
    @LieRyan You may want to edit your answer to exclude the references to "unit tests". – Philip Kendall Jan 31 at 22:48
  • 1
    @LieRyan: “no matter whether it's unit tests or integration tests or system tests, a test should be independently runnable”: no, it's not. System tests have this particularity that some of them may be heavyweight. As I explained in my question, the system test I run interact with an external API: this means that every action takes time, and every action could potentially incur the actual cost of using a paid API. Making things more costly, slow and complex just for the sake of some theoretical principle is never a good thing. – Arseni Mourzenko Jan 31 at 22:59
  • @ArseniMourzenko You are of course correct that many real-world tests cannot be self-contained, quick, and elegant. BUT you're encountering impedance because you're using the unittest framework the wrong way. Lie is suggesting one way to address this: putting all steps of the test scenario into one test. For integration- or system-level tests, I'd rather get rid of the framework and write a completely independent script (but output results for each step in a machine-readable manner, e.g. using the TAP format). I've used this “fail but continue” very frequently in the Perl ecosystem. – amon Feb 7 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.