4

I'm looking for some advice on testing strategies for service to service communication.

I have one service (service A) that makes a call to another service (B) - which is a rest API. Both services are owned by me.

I have some unit tests around the service calls and I simply mock the HTTP library so no requests are actually sent to the service. That works well for unit testing but I was wondering if it is worthwhile to add some integration tests that actual test the service calls and responses.

The problem I see is service B updates a database so any integration tests in service A will have to reset any changes they make by calling the DB directly. To me this doesn't seem ideal as now service A knows more about the implementation of service B than it should.

Are these tests valuable? When I've seen these kind of tests before they are often brittle and rely on development environments being in a good state. If this was a third party API for example I wouldn't have tests which call it directly.

I can think of two options:

  1. Write the integration tests in service A and have these tests call service B's database to reset/inset data as needed.

  2. Stick with mocks and don't add integration tests to service A. Instead add some functional tests to service B which test the various rest endpoints.

Any advice or thoughts?

  • You can mock the service B because the service B should in theory contain integration tests on its own thus verifying its own functionality and thus the service A can simply trust the service B will work according to its given contract. – Andy Sep 17 '17 at 20:27
  • I agree that service B should contain its own integration tests: to test its API and database integration. These don't cover the integration between service A and service B though so it would be assuming (not verifying) that service B maintains its contract as you said. – M.M Sep 18 '17 at 20:58
5

To my experience, our Service Tests should not be bound by dependencies which execution are out of our control.

First of all, let's narrow down the scope of the tests. As stated in the question, the service under test is A, so let's focus on testing A, regardless the ownership of B and its state (running, buggy or not).

One important thing to achieve with tests is determinism. The only way we have to guarantee the correct behaviour of A according to the premises of the (non-)functional requirements is implementing tests that reproduce these premises.

A way for us to achieve determinism is implementing mocks and/or stubss as service A dependencies. We will reproduce the uses cases we need through these. Someone might think that this is not an integration test, but integration tests are not necessarily private from mocks and stubbs. Some integrations might involve several components working together. In these cases, it's good to be able to isolate each of these integrations so that we can check our component's behaviour under several and different integration conditions.

The problem I see is service B updates a database so any integration tests in service A will have to reset any changes they make by calling the DB directly.

Making Service B write in DB is almost anecdotic. Testing against a real Service B we are, indirectly, testing the Service B itself and the whole environment where the service is running at!

The real danger is in the undetermined conditions under which Service B is living at the moment of testing. These conditions, in the worst of the cases, will cause Service A tests to fail. They will fail due to issues unrelated to the code. These fails don't give us meaningful feedback about the state of the code being tested.

Writing in DB is the less of the problems, there are so much more things that could go wrong. For example.

  • Service B has no test environment.
  • Service B has a test environment but has been deployed a new version which includes breaking changes.
  • Service B is buggy.
  • Service B data storage is down or temporarily unavailable.
  • Service B respond with corrupt data.
  • Service B is under test, and the data is frequently changing.
  • Service B is no longer available.

You should be wondering why non-deterministic tests are dangerous. I suggest reading Fowler's blog about Erradicating non-determinism in tests. Take a look at this question too. Doc's answer summarises the subject very well.

An example of non-deterministic tests are Flaky tests. Flaky tests are tests that eventually fail due to undetermined circumstances, causing our tests to fail now and then.

A test suite with flaky tests can become victim of what Diana Vaughan calls normalization of deviance - the idea that over time we can become so accustomed to things being wrong that we start to accept them as being normal and not a problem.

-Building Microservices- by Sam Newman

The normalization of deviance is the seed of the evil.

Sorry, I digress...

Any advice or thoughts?

When testing integrations, neither the data nor the behaviour of the external service should worry you. At least not yet. 1

What should worry you is to test the correct consumption of interface (API) and the proper handling of the feedback (error handling, deserialization, mappings, etc). In other words, the contract.

Lately, I have started to work with the concept of Test Doubles and Consumer Driven Contracts tests with very positive results.

It's true that they require additional efforts addressed to build and maintain these tests. That's our case. However, we have reduced the building, the testing and the deployment time significantly and we get faster and more meaningful feedback from CI.

In the line with the above writing and @Justin's answer, you might be interested in tools like Mountebank.


1: There are a place for tests addressed to validate the real behaviour of the external services. They can be placed out of the building pipeline. They might or might not be essential for a green deployment. That depends on whether you can or not circumvent the issues raised by the service. It's almost a political question rather than technical.

  • 2
    "When testing integrations, neither the data nor the behavior of the external service should worry you." Absolutely not! Assuming that the external service will conform to its contract is dangerously irresponsible, and the whole point of integration testing is to determine whether the complete integration actually behaves as you expect. Yes, it's hard to get this right, it's hard to stop this from being flaky, but your alternative is to accept bugs in production that result from your misunderstanding of the surface area. Testing to the contract has a place - in unit tests, not integration. – closeparen Sep 17 '17 at 21:15
  • One approach I've seen is to support "test tenancy" through both services. To mitigate the concerns you mentioned, you can send rate-limited, test-tenancy-only traffic to the production instance of Service B, and randomly generate new fixtures for each test run. My company operates a whitelist-based proxy to do this safely from dev laptops to production. It's a ton of work to support and propagate test tenancies throughout your infrastructure, but it's less work than a fully duplicated staging environment, and reliability isn't free.You can also easily run local copies of stateless services. – closeparen Sep 17 '17 at 21:21
  • @Closeparen that's why I said When testing integrations, neither the data nor the behavior of the external service should worry you **(not yet)**. The validation of the contract (behavior and possible data) is what we do with Tests Doubles or CDC tests. Basically we separate integrations tests from testing external services. Our deployment pipe line is quite extense. There are unit, integration, CDC tests but also e2e tests and acceptance tests. Additionally we perform out of the pipeline load tests, monkey tests and some manual testing now and then. – Laiv Sep 17 '17 at 21:40
  • For brevety, I wanted to narrow the answer only to integration tests (we call them Service Tests) which don't necessarily need to be performed against real services. They could, but we removed these integrations from the pipeline or we relegated them to other pipelines. The reason is simple, the contract can fail just 1ms after passing your tests so you only got a false possitive. Contracts can change anytime. That should not be a burden for your tests. Your main concern is the correctness of the code you build, maintain and run. – Laiv Sep 17 '17 at 21:47
  • Running local copies of all the services we consume would be the hell on earth. We are a company oriented to services, we work in many different projects with many different integrations. Tests Doubles don't replicate 100% the services they replace, only the surface we are bound to. And only mock the uses cases we need. These are validated against the results obtained from the CDC tests against real services. The thing is that we can detect quickly whether the error is ours or not. If they aren't we decide what to do, but our code should be fully tested all the time – Laiv Sep 17 '17 at 21:57
0

You are right, a test that accesses the database layer directly will be more brittle, however depending on the value the test provides it might still be worth it. e.g. if you are testing an important, fragile piece of functionality in a legacy app, then the value provided by this test may well be worth the cost of maintaining this brittle test.

That said, there are a couple of alternative approaches that you might find useful.

  1. Set up your services / tests so that either the changes can be reset through the API, or so that the changes do not need resetting. For example, if your test creates a user, expose an endpoint that allows you to either delete or deactivate users, or create test users with a prefix that is unique for each test run and ignore those with a different prefix. Remember that testability is a (very valuable) feature of software - you shouldn't feel that it is somehow controversial to introduce features exclusively to improve the testability of your software.

  2. Test against a fake HTTP server, e.g. a mock implementation that records requests received and sends appropriate responses based on the test being run. This has the disadvantage of not testing interaction with the "real" service, however provides coverage that unit tests cannot. In fact this sort of testing can provide coverage of scenarios that can be more difficult against the "real" service, e.g. testing error responses, or high latency.

  • Thanks for the response Justin. The codebase I have inherited is completely untested so I might currently be on the test warpath and I'm aware that creating a whole suite of brittle integration tests might come back to bite us, or it might provide exactly the sort of thing we need. I did consider making API changes to reset the data but I felt making changes to another service just for the purposes of testing didn't seem right. The second idea of a mock HTTP service is an interesting one that I didn't think about! I'll definitely look into that – M.M Sep 16 '17 at 22:13
0

Your integration tests should not involve the DB directly. Well, the question is what interaction you want to test:

+-----+      +-----+      +------+
|  A  |<---->|  B  |<---->|  DB  |
+-----+      +-----+      +------+

Are you trying to test the A–B interaction or the A–B–DB interaction? If you only want to test the A–B subsystem and the B service is some kind of abstraction over the database where no other service is assumed to write to the DB, then you should not access the DB even for your tests.

The most important problem is that B is no longer free to change the database without also updating your A–B integration tests: renaming tables, adding columns, changing the DB technology, etc.

The simplest way to test A–B in isolation is to launch a separate A and B instance only for the test. The DB is effectively a part of B, so you also want to start with a fresh database. It is sometimes feasible to create a new database per test which would be ideal. E.g. creating a new SQLite database is super simple. If setting up a DB is more involved, you can keep a DB around for integration tests that will be reset before each test.

Integration tests are different from unit tests. In unit tests, you want to perform every test in isolation. This is not feasible for integration tests because initialization of the environment is often very involved and time-consuming. It is often best to organize your individual test cases into test suites, where each test case depends on the results of the previous test. The environment is only initialized at the beginning of each test suite. In exchange for faster tests you pay with less useful test results: if an early test case in a test suite fails, the remaining tests can't be run.

One simple way to add integration tests to an existing system is to record real-world interactions, then replay them for the test. E.g. I once wrote a tool that could parse the log output of a system: to create a new testcase I would copy the input & output from the logfile, anonymize the data, and save it to files testname.in.txt and testname.out.txt. The test runner would then go through a directory full of these files, replay the input, and diff the result with the expected output. However, you do have to take care to select representative test cases. Repeating similar tests is a waste of time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.