Interacting with an external server is a major part of the functionality of an app I have to write.
I've always been told that automatic tests are not supposed to interact with the outside world. Instead, enpoints of the app that is being tested are supposed to be mocked out. Thus, if, for example, my app is making calls to a remote upstream API, I am supposed to mock out the library that opens internet connections to this remote API. Now I am supposed to make testing scenarios by checking if my app makes the correct calls to this upstream API given the responses this API (that is, my mock) sends. In no case should running the test suite open connections to the outside world.
So I started writing tests for this app according to the above: I mocked out a HTTP library, put some likely responses from the upstream server into the test data directory, ...
But this is not what my boss wants. Instead, he told me I was supposed to actually allow my app to make calls to the outside world. According to him, if my app must interact with the outside server, the best way to test it is to allow my app to actually interact with this outside server and see if the results are correct.
I was very surprised.
In this way my test suite is dependant on whether the outer server is online and wether the internet connection is working, which I suppose is bad. Another issue is that certain commands that need to be tested can legitimately take a very long time, depending on the answers from the server. If I mocked out this server I could craft the 'answers' from it in such a way that processing them would not take a long time; but my boss wants me to instead make my app process only certain small parts from the server if it is running in test mode to make the test suite run in a reasonable time. This, of course, means that some code paths will have to remain untested forever, since they are not in the certain branch of the
if that checks if this is test mode.
However, I have to grant this: in the way my boss wants, if the upstream server suddenly changes its API, this will be detected by the test suite; if this server was mocked out then such a problem could only be detected at runtime.
Note that this outside server is indeed an outside one; not controlled by our company. The way I understand the principle of integration tests is that there is no mocking or stubbing of our components; but still no automated testing, incl. integration tests, should open connections to the outside world. Even so, I'm not sure if this qualifies as integration testing, since currently there are no tests that do mock out this upstream server, as per my boss' request.
Clearly, there is something here about the principles of testing that I do not understand. Why should automated tests actually open internet connections to the outside world instead of mocking these out?