-2

Into:

There are mutliple ways to test code: unit tests /e2e / manual testing /..

I'm developing a project that it's implementation details changes very quickly (and sometimes the core functions as well).

Some of our microservers talk to each other directly while some others communicate using events like kafka.

Problem:

When I create an e2e test (for the backend image only), before each test(s), I build the docker image of my micro service and run it again on (each) test.

I find it really hard to set up this kind of e2e test for a micro service that directly talks to other microservices (sending get/post/.. requests).

As a result, I also build/pull the other images and run them before each tests as well. But it's not that easy because you can end up implementing a version of docker-compose in you tests infrastructure.

I would like to minimize the amount of errors that can come from other services and test a specific microservice.

Possible solution:

Changing the microservices architecture.

When ever it is possible, a micro service will communicate with others using events. So in the tests, we only need to setup a kafka and the microservice that we try to test.

I only though of this solution from testing perspective and not from "what is best", for example, it's faster to communicate without kafka.

Question:

What are the pros and cons of my proposal? From your experience, is it maintainable?

  • What type of errors are you trying to find? Based on your desire to "test a specific service", end-to-end tests don't sound like the right tool for the job. – Jacob Raihle Jul 13 '20 at 11:43
1

At least one definition of microservices include the following

  • Highly maintainable and testable
  • Loosely coupled
  • Independently deployable

From your description it sounds like you fail all of these criteria, So you might have something more like a distributed monolithic application.

I'm not sure changing calls to events will solve anything. Not all calls can be replaced by events, and you will need to test how the service responds to the events anyway. And even if you create some test framework for the service to interact with, a potential source of problems is in how the services interact. And these kinds of problems will not be found.

One approach would be to fix the problems and reduce the coupling. This should allow you to deploy the service independent from other services and test the service against a API definition that describes the intended behavior. This might be difficult if the implementation changes frequently, but if service API changes frequently then microservices might not be the best approach.

Another approach would be to embrace the monolith and create a test-system that includes all components, possibly with some of the components replaced, or configured specifically for testing. This would ensure that all, or at least most, of the system is tested.

  • 1
    I understand your point. also, monolith is not an option - we need container per service because each container will have different amount of replicas. – Stav Alfi Jul 15 '20 at 6:55
  • 1
    When talking about Monolith, I refer to a system where the components have high coupling. Regardless if the components are individual containerized services, or a single process. The significant difference is if components can be deployed and tested independently or not. – JonasH Jul 15 '20 at 12:26
1

Not every microservice works in isolation, and there are times where you do have to do end-to-end testing. However, I'd like you to rethink how you are doing your testing. It sounds like you are doing both deployment testing and end-to-end testing at once.

  • For deployment testing, consider Terratest (at the very least watch their video: https://youtu.be/xhHOW0EF5u8)
  • You should have a complete environment where you test things. Our team has "integration", "QA", and "Pre-Prod" environments before going to productions.
    • Integration: constantly patched and small tests run to validate obvious things aren't broken
    • QA: updated much less often so that long running end-to-end and UI testing can be performed on what is getting ready to go to production
    • Pre-Prod: Our customer has barriers to deployment we don't like, so we stage what we are planning to give them to make sure the test install works when we go to a new environment
  • Containers start over from scratch every time they are run. There should be no need to build dependencies when testing the effects of updating one microservice.
  • The more you deploy at one time the more likely your deployment can fail (see the Terratest video for the math on that).

Your QA environment should be set up and updated as needed to support your end-to-end, security, integration, and performance/scalability testing. If you are deploying the changes to one microservice, that is the only microservice that gets deployed as part of your build job. You have an independent job that runs the tests in that environment.


Lastly, there are a number of good reasons for introducing events or messaging between microservices. However, not just for the sake of testing. There are a number of companies that learned the hard way that having services call services synchronously only really works in low traffic environments like your QA deployment. Eventing allows the services to implement an eventual consistency policy, which scales much better.

Just remember, scalability is inversely proportional to the amount of communication an action invokes. Asynchronous communication at least helps keep the system stable, even though it might take time for changes to percolate everywhere they need to be.

I highly recommend incorporating scalability testing using something like JMeter or K6 to provide load to your test environment. It might highlight weeknesses in your current architecture.

  • interesting. i have seen that Terratest is for go. We are writing in node. but I will look for something similar. – Stav Alfi Jul 15 '20 at 6:57
  • 1
    Terratest is for deployments. It's just implemented in go. It can be used to test kubernetes, ansible, Terraform, CloudFormation and a number of other deployment mechanisms. – Berin Loritsch Jul 15 '20 at 12:47
  • 1
    The important thing I'd like you to take away from the Terratest video is the corollary of how many things you have to deploy can cause something to go wrong--as well as the mitigation strategies. It's useful information. – Berin Loritsch Jul 15 '20 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.