2

We're in a process of moving from a huge old monolithic architecture towards microservice one. Our product (which is mostly a large single-page app) is quite UI-centric, and thus the current codebase includes lots of automated UI tests.

We expect to have a lot of microservices, each of them updated and deployed dozens times throughout the day, so we have to come up with an automated testing process which allows us to run end-to-end tests against the entire set of services as often as possible. In the perfect world it'd be for every commit in every service, but of course it's not feasible with the our test suite size and available resources.

Our initial idea is to try to have a testing environment which imitates a production environment as much as possible. In that test environment we continuously run end-to-end test suite. Every commit in any service repository (maybe only master branch) causes that service to be built and deployed to that testing environment, and the changes made in that commit will be tested the next time the test suite is run.

When a test suite passes, we can store versions (or commit refs) of each service used at the moment of testing, so that we can keep a database of good and compatible service versions.

This solution has the following downsides: - You won't be sure that your service changes don't break any end-to-end scenarios until the next test suite run passes (can take hours) - It's unclear how to run test suites for changes from branches other than master.

Are there any good working solutions here?

  • 2
    You don't want all your tests to be end-to-end. Only small fraction of all tests should be end-to-end. Majority of tests should be unit tests and tests of each microservice in isolation. IMO, the idea that you test with multiple microservices at the same time is crazy to begin with. It smells like you still have monolith. The idea of microservices is that there are clear and hard boundaries isolating each service, allowing it to be designed, tested and deployed independently of the rest. – Euphoric Sep 5 '17 at 20:44
  • Also, if you are worried about incorrect microservice breaking the whole thing, you should focus on ability to quickly roll-back to version you know works. That would minimize need to test all microservices together, as one faulty microservice can quickly be fixed. – Euphoric Sep 5 '17 at 20:46
  • @Euphoric of course we also have lots of unit and service tests for each particular service, so that it can be tested in isolation from the rest of the system. But we still need to test the integration of all services as close to the end user scenarios as possible. – Andrew Khmylov Sep 5 '17 at 21:02
  • Are you testing your UI for several environments in succession? You might be able to reclaim some time by having those tests run in parallel (one for each browser for instance). – Berin Loritsch Sep 6 '17 at 15:06
4

If the test suite takes literally hours, you should focus on that and try to optimize your tests.

When it comes to testing in a context of SOA, it is important to be able to:

  • Test the internals of the service before deploying it. The tests here will be very granular, white-box tests, which would run in isolation. You will usually have a lot of them, and each one will perform in a matter of milliseconds. The whole set should take a few seconds (in a context of a microservice; for large applications, it may be a few minutes).

  • Test the interfaces (this includes the REST/SOAP interface provided by the service under test, but also how it consumes the underlying services as well; you won't like deploying a service which behaves badly towards other services it uses). Those tests are slightly slower, but still performed in isolation. They may take a few minutes.

Once those tests pass, you may deploy the service to staging environment which is very close to the production environment. Here, more things can be tested:

  • Test the service in interaction with other services. The goal of those tests is to be able to determine whether the interfaces are actually compatible. It should be unnecessary to tell that in order for those tests to have any value, the other services should match exactly the versions deployed in production. Those tests can take a few minutes.

  • Test a few end-to-end scenarios to ensure the whole chain works well. Chose the scenarios carefully: since each test may take up to a minute, you don't want too much tests here. There is no need to test every possible case, since you should already have enough coverage (and confidence) with all the other tests performed previously.

Now that you're confident that the service behaves well with the services deployed in production, the service can be pushed on some production machines. From this moment:

  • Test how the service is actually running. Keep it running for five minutes on the selected servers, and check whether the number of errors in the logs have increased or if the other services have a peak of demands, or other things which could indicate that something is not right. If this happens, the service should be rolled back.

  • Test a few end-to-end scenarios which are the most important ones, in order to be sure that the whole chain works in production as well. For instance, for an e-commerce website, a single test which registers a user, adds a product to a cart, makes a purchase, pays, then asks for a refund is largely enough: you don't need to make an additional test to know if the user can unregister or if a user can compare products—those scenarios are too minor and should have been already tested previously.

Once the system is confident that the service runs well in production, it may be deployed on the remaining production machines.

It's unclear how to run test suites for changes from branches other than master.

This depends entirely on your continuous integration strategy. In many cases, you do run unit tests on branches, but there is no continuous integration, which means that the code from branches is never pushed to staging environment. This, in turn, encourages the team to integrate often, either by not using branches at all, or by merging them on regular basis (that is, several times per day, or at least once per day).


Note that the specificity of a microservices ecosystem is that your product is not the whole set of services; your product is the specific service. This means that when you change its implementation, the only thing you should care about is that you haven't changed its interface; as soon as the interface is not changed, all the services which use the service you modified are expected to work correctly.

In fact, you don't even have to know which services are using yours.

In the same way, when Twilio or Amazon are changing their services, they won't run any tests to ensure that an application you have written which uses their services still works. Similarly, you don't test your app every time Amazon redeploys S3 (nor would you even know they redeployed it).

This also means that a lot of your effort should be spent at carefully designing interfaces:

  • Which won't need to constantly change.
  • Which will be detailed, unambiguous and well documented.
  • Which won't leak implementation.
0

The point of Continuous Integration/Deployment is to provide feedback as quickly as possible. If your tests take 2 hours to run, then it strikes me as something where your automated environment needs to be optimized:

  • Look for ways to minimize "ripple effect" of one service being updated
  • The build should allow things to run in parallel that don't directly depend on each other
  • UI tests could be put into packages so that when one service changes, the affected UI test packages run

In other words, you want the sanity checks to be known within 5-10 minutes at the maximum. Full test suites can be run at regular intervals, allowing for parallelization whenever possible.

The more confidence you have that your microservices are well contained and designed, the less you have to retest already known working code. Avoiding unnecessary retesting will shorten the cycle time to know when something broke something else.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.