To my experience, our Service Tests should not be bound by dependencies which execution are out of our control.
First of all, let's narrow down the scope of the tests. As stated in the question, the service under test is A, so let's focus on testing A, regardless the ownership of B and its state (running, buggy or not).
One important thing to achieve with tests is determinism. The only way we have to guarantee the correct behaviour of A according to the premises of the (non-)functional requirements is implementing tests that reproduce these premises.
A way for us to achieve determinism is implementing mocks and/or stubss as service A dependencies. We will reproduce the uses cases we need through these. Someone might think that this is not an integration test, but integration tests are not necessarily private from mocks and stubbs. Some integrations might involve several components working together. In these cases, it's good to be able to isolate each of these integrations so that we can check our component's behaviour under several and different integration conditions.
The problem I see is service B updates a database so any integration
tests in service A will have to reset any changes they make by calling
the DB directly.
Making Service B write in DB is almost anecdotic. Testing against a real Service B we are, indirectly, testing the Service B itself and the whole environment where the service is running at!
The real danger is in the undetermined conditions under which Service B is living at the moment of testing. These conditions, in the worst of the cases, will cause Service A tests to fail. They will fail due to issues unrelated to the code. These fails don't give us meaningful feedback about the state of the code being tested.
Writing in DB is the less of the problems, there are so much more things that could go wrong. For example.
- Service B has no test environment.
- Service B has a test environment but has been deployed a new version which includes breaking changes.
- Service B is buggy.
- Service B data storage is down or temporarily unavailable.
- Service B respond with corrupt data.
- Service B is under test, and the data is frequently changing.
- Service B is no longer available.
You should be wondering why non-deterministic tests are dangerous. I suggest reading Fowler's blog about Erradicating non-determinism in tests. Take a look at this question too. Doc's answer summarises the subject very well.
An example of non-deterministic tests are Flaky tests. Flaky tests are tests that eventually fail due to undetermined circumstances, causing our tests to fail now and then.
A test suite with flaky tests can become victim of what Diana Vaughan calls normalization of deviance - the idea that over time we can become so accustomed to things being wrong that we start to accept them as being normal and not a problem.
-Building Microservices- by Sam Newman
The normalization of deviance is the seed of the evil.
Sorry, I digress...
Any advice or thoughts?
When testing integrations, neither the data nor the behaviour of the external service should worry you. At least not yet. 1
What should worry you is to test the correct consumption of interface (API) and the proper handling of the feedback (error handling, deserialization, mappings, etc). In other words, the contract.
Lately, I have started to work with the concept of Test Doubles and Consumer Driven Contracts tests with very positive results.
It's true that they require additional efforts addressed to build and maintain these tests. That's our case. However, we have reduced the building, the testing and the deployment time significantly and we get faster and more meaningful feedback from CI.
In the line with the above writing and @Justin's answer, you might be interested in tools like Mountebank.
1: There are a place for tests addressed to validate the real behaviour of the external services. They can be placed out of the building pipeline. They might or might not be essential for a green deployment. That depends on whether you can or not circumvent the issues raised by the service. It's almost a political question rather than technical.