Test multiple versions in acceptance

Question

Introduction I'm working at a software company building​ software as a service. Currently we have version x.y in production and x.z on acceptance. Let's say we find a critical bug on production and want to test the fix on an acceptance environment. I don't know how to work with this.

Options

1. Create a second acc environment for x.y
2. Overwrite the current acceptance environment with x.y
3. Fix it in x.z and deploy that asap to acc and prod
4. Other

For me option 1 feels the safest, but I don't see anyone else doing that (maybe I'm not looking hard enough). How do other people do this? Is there any documentation about this?

Answer 1

This all depends how easy it is to change the acceptance environment.

If deployment on acceptance environment is easy...

...for example if you just have to replace the executables, you may consider option 2:

• overwrite current acceptance
• make the test
• deploy the corrected version
• redeploy (or rollback to) the current version under development
• continue acceptance development

If deployment on acceptance is more delicate...

Unfortunately many complex business applications are more delicate to deploy: you may need to install a set of dependent dynamic libraries or plugins, or you might need to update some configuration files, etc.... This makes it risky to move back and forth.

The showstopper is when there is a database, and different version might cope differently with the persistent data. Overriding acceptance could in this case lead to persistent inconsistencies.

Therefore the safest approach is to go for option 1. This is the standard procedure for the upgrade of major ERP systems, when the whole company depends on the production environment. There, the clone acceptance is even created at the beginning of the upgrade, just in case of...

This approach is also the most expensive one. But if your company has a professional datacenter, they should be able to virtualize such an environment and restore older images very easily and without huge overhead costs.

If the risks are low...

Last but not least, a risk assessment could consider that the patch is a minor change with no real risk. In this case, an exception procedure could lead to a direct development (option 3). But even if it's a known practice, I can't recommend this for critical systems, except in rare exceptions where there is no other alternative.

Answer 2

This is always a question of money. If it isn't a question about money, frame it terms of money anyway.

If your cost of making an environment is low, e.g. spinning up dozens of virtual machines from build scripts, then make them whenever needed. The cost is low and pays for the benefits.

If you cost of making an environment is high, e.g., mount a spare scratch-monkey or embed in a fresh aircraft carrier, then off course you won't. You choose "fix it in x.z" because that's the obvious answer.

Outside of the extreme cases, having some wild guess of "how much does an environment build cost", "how much does this bug cost per day", and "how much does it cost for a client to be down" can greatly improve your choice of tradeoffs.

The equations change as your business changes. A common progression for software services is to hack on it live when you have a few experimenters on your alpha, auto deploy from GIT on your first paying customers, then add deploy only after testing passes, then add deploy with automatic rollback on crashes, then add progressive deploys with automatic rollback on decline of key metrics. The extra care and cost is justified at each step by the higher cost of outages.

Reason with numbers. Preferably dollars.

Answer 3

You're gona need more environments.

I've worked in places with 50+ test environments; which was a bit overkill in my opinion, but it worked for them.

I think you need at least 2 if you are continuously developing features. 1 to continuously deploy and test the dev version and 1 to test releases, hotfixes, backwards compatibility, stress testing etc while the development rolls on.

Then you prob also want a demo environment the sales guys can use to show to prospective client, Maybe one with mocked services so you can run ui tests quickly?

Once you have automated deployments to cloud infrastructure the temptation is always to add more and more! But remember, someone has to maintain them all and that can be a pain in the arse