Where do I run my integration tests

Question

We currently have 4 environments for a project (local/dev, test, acceptance, production). We created some integration tests which want to run. The big question we now have is where to run them?

1. Run them on all environments. The pro's for this is that we know for sure it works on all environments. The cons are that a) you don't want to stress your production environment with tests, b) Not all tests can be preformed on production since the external references from third parties of also production

2. Run them on single environment. So you pick one of the four. The problem with this is that environment specific settings (database, config) can cause bugs. So if you tests run fine on the test environment bug due to a config bug in production it breaks.

3. Create a production environment on the fly overwrite only the settings you really need for the test and run your tests on those. The pros of this approach is that for production you filter out the environment specific settings bugs as much as possible. They can still occur on the other environment, but the consequences are less. The cons are that this approach takes quite some time to set up, can it be skipped by minimized the differences between the environments..

I'm wondering, is there a standard for this? What would normally be done?

Answer 1

You'd run them on the test/acceptance environment. The acceptance environment is supposed to mimic production (as much as possible). The only differences should be configuration and possibly amount of servers (this depends on how you deploy). I use automated tools which validate config (and fails the deployment before it starts or during it), maybe you could use a similar approach. My deployment tool (Octopus Deploy) can handle rollbacks.

Sidenote 1: You should never write tests against production nor should you clone your production environment. You could try using a Blue Green Deployment method and this would mean you'd be able to sense checkwhat is deployed in your inactive production environment before switching your load balancer(s) over. Possibly even run some validation tools for whatever has been deployed.

Sidenote 2: I don't know why you have both test and acceptance environments, unless this is some sort of business requirement but I'd run integration tests on both. We have, development, quality (acceptance) and production. Our integration tests (database connectivity etc) run on all but production. Our acceptance tests run on quality (acceptance). You could run your integration tests locally if you have all of the neccessary systems installed locally i.e. if you're connecting to SQL, Mongo, Redis but for us that's not possible.