Should automatic tests interact with the outer world?

Question

Interacting with an external server is a major part of the functionality of an app I have to write.

I've always been told that automatic tests are not supposed to interact with the outside world. Instead, enpoints of the app that is being tested are supposed to be mocked out. Thus, if, for example, my app is making calls to a remote upstream API, I am supposed to mock out the library that opens internet connections to this remote API. Now I am supposed to make testing scenarios by checking if my app makes the correct calls to this upstream API given the responses this API (that is, my mock) sends. In no case should running the test suite open connections to the outside world.

So I started writing tests for this app according to the above: I mocked out a HTTP library, put some likely responses from the upstream server into the test data directory, ...

But this is not what my boss wants. Instead, he told me I was supposed to actually allow my app to make calls to the outside world. According to him, if my app must interact with the outside server, the best way to test it is to allow my app to actually interact with this outside server and see if the results are correct.

I was very surprised.

In this way my test suite is dependant on whether the outer server is online and wether the internet connection is working, which I suppose is bad. Another issue is that certain commands that need to be tested can legitimately take a very long time, depending on the answers from the server. If I mocked out this server I could craft the 'answers' from it in such a way that processing them would not take a long time; but my boss wants me to instead make my app process only certain small parts from the server if it is running in test mode to make the test suite run in a reasonable time. This, of course, means that some code paths will have to remain untested forever, since they are not in the certain branch of the if that checks if this is test mode.

However, I have to grant this: in the way my boss wants, if the upstream server suddenly changes its API, this will be detected by the test suite; if this server was mocked out then such a problem could only be detected at runtime.

Note that this outside server is indeed an outside one; not controlled by our company. The way I understand the principle of integration tests is that there is no mocking or stubbing of our components; but still no automated testing, incl. integration tests, should open connections to the outside world. Even so, I'm not sure if this qualifies as integration testing, since currently there are no tests that do mock out this upstream server, as per my boss' request.

Clearly, there is something here about the principles of testing that I do not understand. Why should automated tests actually open internet connections to the outside world instead of mocking these out?

Answer 1

Should automatic tests interact with the outer world?

Some of them, yes.

We typically run different suites of tests at different times in the product life cycle.

If what you are trying to measure is how your code behaves in production, the most accurate test is going to be a test against that code running in the production environment. Slightly less accurate is going to be testing that code running in a copy of the production environment, or in a staging environment.

But! Testing that way has lousy investment odds, in that many kinds of mistakes can be found using more cost effective testing techniques. For instance, we can catch a lot of errors in our code logic by running tests in simulated envionrments.

Furthermore, with some care and thinking we can choose designs that separate our complicated and risky logic from the details of the environment that it runs in. So we perform a bunch of inexpensive tests in simulated (mocked) environments until we are confident that our code is behaving the way we intend, and then measure how well this properly behaving code works with the "real" environments.

If we are resource constrained, we then start thinking about which trade offs we should make. Your boss's preference for testing against real systems makes a lot of sense to me if that is where, historically, the problems have been -- especially if the real systems have been unstable.

An interesting idea to explore is that of "contract testing" - the contract acts as a natural seam between the real system and your work. In effect, you write automation that checks your assumptions about how the remote system works (so that you get a warning when that changes unexpectedly), and you write tests to make sure that your code works if the remote system works the way that you assume (cheaper than testing live all the time), with the idea being that there should be very few mistakes to catch when you put the two systems together.

Answer 2

The purpose of testing is to mitigate risk. Testing with mocks or stubs can mitigate the risk that your team has introduced a code flaw. Testing with the actual service can mitigate the risk that the overall solution no longer works. In most real-world projects, there is a practical need for both.

Answer 3

Unit tests shouldn't interact with the outside world. The best quality of unit tests tests is that they should be fast, and interacting with the outside world isn't going to be fast. The majority of your tests, especially tests that tests for error handling and corner cases should be a unit test.

However, not all automated tests are unit tests. If you're connecting to third party APIs, you should also have an automated tests that make an actual connections to a real server, either with a dedicated test account on the target APIs production server or on their staging server. If these live testing are significantly slower than the rest of your tests, you may want to consider setting up these live tests so they only run when explicitly requested.

The problem with unit tests is that unit tests only tests your implementations against your assumptions of how the API behave; and your assumptions are formed usually by reading documentations or some manual prodding to see how the service behaves. I've seen way too many instances when doing integrations where API documentations aren't in line with their actual implementation, or the APIs docs are unclear or unambiguous, and you ended up implementing the API in a way that is not supported by the implementation, and which just flat out doesn't work or breaks later on when the API provider make future updates.

Automating live testing simplifies QA, as it allows you to quickly check that the basic functionalities of your integration still actually work.

I generally keep these live testing to a minimum, usually testing only the "happy path", as setting up to generate errors in real server are often fairly tricky. There's usually a diminishing return in any testing, and live testing in particular diminishes rapidly beyond the happy paths.

Answer 4

Should automatic tests interact with the outer world?

That rather depends on what type of automatic tests you're talking about. If they're unit tests, then no - they should be self contained following FIRST principles.

As for whether you write other types of tests e.g. integration tests - well, think very carefully about this. They tend to be brittle, hard to write, difficult to verify, and are long running. There is a school of thought as to whether these are worth writing at all. I think this is a bit of an extreme view, but the general premise is worth considering - see if the testing can be done in a more cost efficient way.

However, they can be cheaper than deploying each time and smoke testing manually. Like many things, it is a trade off. If you can prove you have a working system by writing only a few of these integration tests, it may be worthwhile. If you're still getting lots of failures despite these tests, then you need to investigate other avenues.