I am writing tests for a project that consists of multiple submodules. Each test case that I have written runs independent of each other and I clear all data between tests.

Even though the tests run independently, I am considering enforcing an execution order, as some cases require more than one submodule. For example, a submodule is generating data, and another one is running queries on the data. If the submodule generating the data contains an error, the test for the query submodule will also fail, even if the submodule itself works fine.

I can not work with dummy data, as the main functionality I am testing is the connection to a black box remote server, which only gets the data from the first submodule.

In this case, is it OK to enforce an execution order for the tests or is it bad practice? I feel like there is a smell in this setup, but I can not find a better way around.

edit: the question is from How to structure tests where one test is another test's setup? as the "previous" test is not a setup, but tests the code which performs the setup.

  • 3
    Possible duplicate of How to structure tests where one test is another test's setup? – gnat Apr 3 at 13:33
  • 121
    If you're testing the connection to a remote server, then they're by definition not unit tests. – Telastyn Apr 3 at 13:34
  • 9
    The first answer confused me here because in your title you said "Is it bad practice?" and in the summary of your question you wrote "is it ok?" Anyone answering either yes or no is going to confuse one of those! – Liath Apr 3 at 13:53
  • 8
    Sounds like you are creating set of integration tests. Even for that one test shouldn't rely on other tests. – Low Flying Pelican Apr 3 at 14:03
  • 16
    If the order matters then you are probably doing it wrong. – Mark Rogers Apr 3 at 15:39
up vote 234 down vote accepted

I can not work with dummy data, as the main functionality I am testing is the connection to a black box remote server, which only gets the data from the first submodule.

This is the key part for me. You can talk about "unit tests" and them "running independently of each other", but they all sound like they are reliant on this remote server and reliant on the "first sub module". So everything sounds tightly coupled and dependent on external state. As such, you are in fact writing integration tests. Having those tests run in a specific order is quite normal as they are highly dependent on external factors. An ordered test run, with the option of an early quit out of the test run if things goes wrong is perfectly acceptable for integration tests.

But, it would also be worth taking a fresh look at the structure of your app. Being able to mock out the first submodule and external server would then potentially allow you to write true unit tests for all the other submodules.

  • 14
    Not to mention that some test has to specifically check that the expected behaviour occurs when the remote server is unavailable. – Alexander Apr 4 at 8:56
  • 2
    Or, perhaps, you are in fact intending to write integration tests, and thus mocking out the data is not going to achieve what you are trying to accomplish with these tests. – Guy Schalnat Apr 4 at 11:51
  • 10
    Problem is most likely that Junit has "unit" in its name. – Thorbjørn Ravn Andersen Apr 5 at 8:48
  • 6
    @ThorbjørnRavnAndersen Exactly. People naturally write integration tests, rather than unit tests, because integration tests are both far more useful and far less difficult to write than "real" unit tests. But because the popular testing frameworks were named after the concept of unit tests, the term has been co-opted and come to mean "any automated testing" in modern parlance. – Mason Wheeler Apr 5 at 21:24
  • 1
    @MasonWheeler Or even co-opted by non technical managers to mean acceptance testing. – TKK Apr 6 at 0:02

Yes, it's a bad practice.

Generally, a unit test is intended to test a single unit of code (e.g. a single function based on a known state).

When you want to test a chain of events that might happen in the wild, you want a different test style, such as an integration test. This is even more true if you're depending on a third party service.

To unit test things like this, you need to figure out a way to inject the dummy data, for example implementing a data service interface that mirrors the web request but returns known data from a local dummy data file.

  • 8
    Agreed. I think this confusion stems from the fact that that many people have an idea that integration tests have to be end-to-end, and use "unit test" to refer to any test that only tests one layer. – autophage Apr 3 at 13:56
  • 8
    @autophage: Definitely agree with this. In fact I agree with it so much that I regularly find myself falling into the same trap despite agreeing that it is a trap 😂 – Lightness Races in Orbit Apr 3 at 16:16

The enforced execution order you propose only makes sense if you also abort the test run after the first failure.

Aborting the test run on the first failure means that each test run can uncover only a single problem and it can't find new problems until all preceding problems have been fixed. If the first test to run finds a problem that takes a month to fix, then during that month effectively no tests will be executed.

If you don't abort the test run on the first failure, then the enforced execution order doesn't buy you anything because each failed test needs to be investigated anyway. Even if only to confirm that the test on the query submodule is failing because of the failure that was also identified on the data generating submodule.

The best advice I can give is to write the tests in such a way that it is easy to identify when a failure in a dependency is causing the test to fail.

The smell you're referring to is the application of the wrong set of constraints and rules to your tests.

Unit Tests often get confused with "automated testing", or "automated testing by a programmer".

Unit Tests must be small, independent, and fast.

Some people incorrectly read this as "automated tests written by a programmer must be small independent and fast". But it simply means if your tests are not small, independent, and fast, they are not Unit Tests, and therefore some of the rules for Unit Tests should not, cannot, or must not apply for your tests. A trivial example: you should run your Unit Tests after every build, which you must not do for automated tests that are not fast.

While your tests not being Unit Tests means you can skip one rule and are allowed to have some interdependence between tests, you also found out that there are other rules which you may have missed and will need to reintroduce - something for the scope of another question.

As noted above, what you are running seems to be an integration test, however you state that:

For example, a submodule is generating data, and another one is running queries on the data. If the submodule generating the data contains an error, the test for the query submodule will also fail, even if the submodule itself works fine.

And this may be a good place to start refactoring. The module that runs queries on the data should not be dependent on a concrete implementation of the first (data generating) module. Instead it would be better to inject an interface containing the methods to get to that data and this can then be mocked out for testing the queries.

e.g.

If you have:

class Queries {

    int GetTheNumber() {
        var dataModule = new Submodule1();
        var data = dataModule.GetData();
        return ... run some query on data
    }
}

Instead prefer:

interface DataModule {
    Data GetData();
}


class Queries {

    IDataModule _dataModule;

    ctor(IDataModule dataModule) {
       _dataModule = dataModule;
    }

    int GetTheNumber() {
        var data = _dataModule.GetData();
        return ... run some query on data
    }
}

This removes the dependency from queries on your data source and allows you to set up easily repeatable unit tests for particular scenarios.

The other answers mention that ordering tests is bad (which is true most of the time), but there is one good reason for enforcing order on test execution: making sure your slow tests (i.e., integration tests) run after your faster tests (tests that don't rely on other outside resources). This ensures that you execute more tests faster, which can speed up the feedback loop.

  • 2
    I'd be more inclined to investigate why a certain unit test is running slowly rather than enforce an order. Unit tests are supposed to be fast. – Robbie Dee Apr 4 at 14:27
  • The OP says he cannot work with dummy data for some of these tests. That means a database hit of some sort, slowing down all tests (even some true unit tests that should run fast naturally). If he has other tests that do not require database hits, they will run an order of magnitude faster than anything requiring disk or network hits. – Mike Holler Apr 4 at 15:46
  • 2
    You're both right, I think; Robbie is right that unit tests should be small and fast and isolated from dependencies so order shouldn't matter and random-ordering often encourages better design by enforcing that independence; and Mike is right that running faster tests first is very, very good for integration tests. As in answers above, part of the problem is terminology of unit vs. integration tests. – WillC Apr 4 at 23:39
  • @MikeHoller Then they're not unit tests. There really should be no confusion as to what unit tests are. – Robbie Dee Apr 5 at 7:35
  • @RobbieDee I was merely using the terminology the OP was using. I understand that these are not true unit tests. If you want to fight about terminology, bring it up with OP. (hence why I clarified with "true unit tests" in my earlier comment") – Mike Holler Apr 5 at 15:31

protected by gnat Apr 4 at 15:27

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.