We are writing a cloud application in a micro-service architecture. We have good unit and integration test coverage for the individual services and we have a set of (public) API level acceptance tests.
For simplicity I will use a contrived example to step through the problem at hand. Let's say I have a Company and User resources and I want to test that User A has access to Company A's user's list but cannot list Company B's users.
I can take one of the two basic approaches:
- Assume that I have a state in the system that is conducive to running this particular scenario
- Bootstrap the state using the public APIs.
I dislike the first approach because now I have to have the same test companies and test users in all environments bootstrapped ahead of time with some script or manually. Failing tests can leave the environment in an inconsistent state, changing the state to ensure I can test a new scenario can effect other tests that I did not touch. In my experience such tests are extremely hard to maintain especially as you grow or tweak existing functionality.
For our acceptance tests we went with the second approach. For my example, I would create two new (random) Company resources, I would create random users under those companies and then make requests to ensure that users in one company cannot see users in the other. This technique has many benefits. It does not assume any previous state in any environment, so I can run them almost anywhere with little effort, tests do not effect each other, I can parallelize such tests very easily because they always run in their own 'sandbox'. Writing tests this way is also encouraged by the book Continuous Delivery.
However we ran into a little snag. Our APIs (currently) allow for creating data in the system but not so much cleaning it up. There is certain data that gets 'hard deleted' but we are required to keep some data around and only 'soft delete' it.
In 'lower' environments this is not a big deal but as we get closer to production environment it is a harder sell that we 'pollute' the environment so much.
Do we need to introduce special APIs to clean up (hard delete) data but may only be used by tests? Do I need to write a separate test suite for production environment that does assume a certain state and maybe changes the state of existing resources but does not add more data (limited level of testing). Do I only run acceptance tests during the deploy window and restore the database to the previous state (may not work with a blue/green deployment scenario)?
The requirement from management is that they want acceptance tests to run against THE production environment frequently to show the system is still in an acceptable condition so we can see/fix issues before a user would see them. I personally feel like it should be enough to run acceptance tests after deployment because the versions of services and their configuration does not change between deployments. The only issues that can/should happen between deployments is that a part of the system 'goes down', like running out of DB connections, disk space filling up, etc... those should be detectable by 'non-functional' health-check type of diagnostics. Am I wrong?