2

We are writing a cloud application in a micro-service architecture. We have good unit and integration test coverage for the individual services and we have a set of (public) API level acceptance tests.

For simplicity I will use a contrived example to step through the problem at hand. Let's say I have a Company and User resources and I want to test that User A has access to Company A's user's list but cannot list Company B's users.

I can take one of the two basic approaches:

  1. Assume that I have a state in the system that is conducive to running this particular scenario
  2. Bootstrap the state using the public APIs.

I dislike the first approach because now I have to have the same test companies and test users in all environments bootstrapped ahead of time with some script or manually. Failing tests can leave the environment in an inconsistent state, changing the state to ensure I can test a new scenario can effect other tests that I did not touch. In my experience such tests are extremely hard to maintain especially as you grow or tweak existing functionality.

For our acceptance tests we went with the second approach. For my example, I would create two new (random) Company resources, I would create random users under those companies and then make requests to ensure that users in one company cannot see users in the other. This technique has many benefits. It does not assume any previous state in any environment, so I can run them almost anywhere with little effort, tests do not effect each other, I can parallelize such tests very easily because they always run in their own 'sandbox'. Writing tests this way is also encouraged by the book Continuous Delivery.

However we ran into a little snag. Our APIs (currently) allow for creating data in the system but not so much cleaning it up. There is certain data that gets 'hard deleted' but we are required to keep some data around and only 'soft delete' it.

In 'lower' environments this is not a big deal but as we get closer to production environment it is a harder sell that we 'pollute' the environment so much.

Do we need to introduce special APIs to clean up (hard delete) data but may only be used by tests? Do I need to write a separate test suite for production environment that does assume a certain state and maybe changes the state of existing resources but does not add more data (limited level of testing). Do I only run acceptance tests during the deploy window and restore the database to the previous state (may not work with a blue/green deployment scenario)?

The requirement from management is that they want acceptance tests to run against THE production environment frequently to show the system is still in an acceptable condition so we can see/fix issues before a user would see them. I personally feel like it should be enough to run acceptance tests after deployment because the versions of services and their configuration does not change between deployments. The only issues that can/should happen between deployments is that a part of the system 'goes down', like running out of DB connections, disk space filling up, etc... those should be detectable by 'non-functional' health-check type of diagnostics. Am I wrong?

  • Companies that shifted to MS, often craft their own tools for dealing with situations like this. You could deploy a private API which main purpose is populate the DBs with the dummy data and hard delete it. I said API but It could be virtually anything. From web APIs to shell scripts. You will find that most of the tools out there only meet your needs gradually. So don't be astonished if you come to the conclusion that the best you can do is to craft more software. – Laiv Sep 3 '17 at 18:39
  • @Laiv I agree to a point. However everyone's first instinct should be to use well established tools and practices first before rolling their own. MS is already extremely complicated. – c_maker Sep 10 '17 at 12:04
1

Add cleanup 'APIs'

But theres no need to make them available. It could be that you just have a SQL script to remove the test data.

Another approach is to just keep a couple of test companies around and reuse them. But This really just pushes the problem down a level, you now have to clean up 'users' or whatever.

Plus you have the additional problems of tests interfering with each others data and the companies becoming corrupted.

Being able to run tests on Prod is very important. Users report problems which may or may not be bugs. You want to be able to run a check to make sure things are working as expected.

  • Being able to run tests on Prod is very important. -- For those production problems that can be only reproduced using someone's production data. For everything else, it's better to have a test or staging environment that functionally duplicates the production environment. – Robert Harvey Sep 3 '17 at 15:46
  • well they are also important too. But no, prod environments fail for all sorts of reasons. You need to be able to test them – Ewan Sep 3 '17 at 15:48
  • 1
    "orders aren't as high as I would expect/like" = run 1000 test orders to make sure its all working – Ewan Sep 3 '17 at 15:49
1

I think first thing you should try is pushing back "the requirement from management". I don't see the point running whole acceptance tests in production. The garbage data is not only the problem caused by the tests in production. How about the bugs introduced by your tests? Another problem is the fear of running tests in production, developers will think twice before putting test in this category. Then you are risking getting less and less acceptance tests.

There are some tech giants do some kinds tests in production. E.g. Netflix runs "Failure Injection Testing" in production. They even open sourced chaos monkey. But these level of tests are simulate different kinds of failure in production to verify the resilience of services. I can see the point of running such tests in production. These tests covers gaps between your production and UAT environments - such as activity load or expensive load balancer, etc.

Rather, we should try making the systems more traceable and monitorable. Theoretically, the environments in your deployment pipeline are getting closer and closer to your production when you go through it.

You've added the arguments for "Being able to run tests on Prod is very important." by saying

For those production problems that can be only reproduced using someone's production data.

This is actually a little bit contradict to the "sandbox" acceptance test idea. The tests should be written in a manner of not relying any existing state. However, in your comment, you do rely on the existing data to reproduce issue.

If you really can't push that back, you should try to get a limited scope on these tests which just covering the gap between your UAT and production.

All resorts mentioned before don't fit your situation, and you still want to run acceptance tests in production, then the no state dependency strategy you've mentioned is the right track to go. Probably, it's also beneficial for you to introducing something like co-relation id on your entities/aggregates created by your tests and your live system may not showing them to real user. Again, be careful with this approach as it introduces another complexity on your production code.

  • "For those production problems that can be only reproduced using someone's production data." was a comment by someone else, not me. – c_maker Sep 10 '17 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.