What are techniques to run acceptance tests in a cloud app without polluting the environment with test data?

Question

We are writing a cloud application in a micro-service architecture. We have good unit and integration test coverage for the individual services and we have a set of (public) API level acceptance tests.

For simplicity I will use a contrived example to step through the problem at hand. Let's say I have a Company and User resources and I want to test that User A has access to Company A's user's list but cannot list Company B's users.

I can take one of the two basic approaches:

1. Assume that I have a state in the system that is conducive to running this particular scenario
2. Bootstrap the state using the public APIs.

I dislike the first approach because now I have to have the same test companies and test users in all environments bootstrapped ahead of time with some script or manually. Failing tests can leave the environment in an inconsistent state, changing the state to ensure I can test a new scenario can effect other tests that I did not touch. In my experience such tests are extremely hard to maintain especially as you grow or tweak existing functionality.

For our acceptance tests we went with the second approach. For my example, I would create two new (random) Company resources, I would create random users under those companies and then make requests to ensure that users in one company cannot see users in the other. This technique has many benefits. It does not assume any previous state in any environment, so I can run them almost anywhere with little effort, tests do not effect each other, I can parallelize such tests very easily because they always run in their own 'sandbox'. Writing tests this way is also encouraged by the book Continuous Delivery.

However we ran into a little snag. Our APIs (currently) allow for creating data in the system but not so much cleaning it up. There is certain data that gets 'hard deleted' but we are required to keep some data around and only 'soft delete' it.

In 'lower' environments this is not a big deal but as we get closer to production environment it is a harder sell that we 'pollute' the environment so much.

Do we need to introduce special APIs to clean up (hard delete) data but may only be used by tests? Do I need to write a separate test suite for production environment that does assume a certain state and maybe changes the state of existing resources but does not add more data (limited level of testing). Do I only run acceptance tests during the deploy window and restore the database to the previous state (may not work with a blue/green deployment scenario)?

The requirement from management is that they want acceptance tests to run against THE production environment frequently to show the system is still in an acceptable condition so we can see/fix issues before a user would see them. I personally feel like it should be enough to run acceptance tests after deployment because the versions of services and their configuration does not change between deployments. The only issues that can/should happen between deployments is that a part of the system 'goes down', like running out of DB connections, disk space filling up, etc... those should be detectable by 'non-functional' health-check type of diagnostics. Am I wrong?

Answer 1

Add cleanup 'APIs'

But theres no need to make them available. It could be that you just have a SQL script to remove the test data.

Another approach is to just keep a couple of test companies around and reuse them. But This really just pushes the problem down a level, you now have to clean up 'users' or whatever.

Plus you have the additional problems of tests interfering with each others data and the companies becoming corrupted.

Being able to run tests on Prod is very important. Users report problems which may or may not be bugs. You want to be able to run a check to make sure things are working as expected.

Answer 2

I think first thing you should try is pushing back "the requirement from management". I don't see the point running whole acceptance tests in production. The garbage data is not only the problem caused by the tests in production. How about the bugs introduced by your tests? Another problem is the fear of running tests in production, developers will think twice before putting test in this category. Then you are risking getting less and less acceptance tests.

There are some tech giants do some kinds tests in production. E.g. Netflix runs "Failure Injection Testing" in production. They even open sourced chaos monkey. But these level of tests are simulate different kinds of failure in production to verify the resilience of services. I can see the point of running such tests in production. These tests covers gaps between your production and UAT environments - such as activity load or expensive load balancer, etc.

Rather, we should try making the systems more traceable and monitorable. Theoretically, the environments in your deployment pipeline are getting closer and closer to your production when you go through it.

You've added the arguments for "Being able to run tests on Prod is very important." by saying

For those production problems that can be only reproduced using someone's production data.

This is actually a little bit contradict to the "sandbox" acceptance test idea. The tests should be written in a manner of not relying any existing state. However, in your comment, you do rely on the existing data to reproduce issue.

If you really can't push that back, you should try to get a limited scope on these tests which just covering the gap between your UAT and production.

All resorts mentioned before don't fit your situation, and you still want to run acceptance tests in production, then the no state dependency strategy you've mentioned is the right track to go. Probably, it's also beneficial for you to introducing something like co-relation id on your entities/aggregates created by your tests and your live system may not showing them to real user. Again, be careful with this approach as it introduces another complexity on your production code.