Running automation tests in production

Question

Although many people understand that even if we have the same setup for TEST and UAT environments as we have for PROD, there're still a lot of reasons why it makes sense to run smoke tests in production:

• Ensure stability of the core functionality
• Identify potential bugs and fix them right after deployment

We'd like to run automated tests cases(not all of them, just those that check core business features) every time when deployment completed.

Everything looks fine except one fact that any tests(manual and automated) leave some rubbish data and we cannot leave it there, because it might impact some business features (for instance reporting). From my point of view there're the following strategies:

1. Clean data created by your tests. Doesn't look perfect due to the fact that any mistake in the process can delete real production data
2. Switch to another database schema to execute tests and switch back to real production schema. This approach requires additional DevOps effort and might introduce additional downtime(SLA will be lower)

Could you please advice any best practices of running tests in production?

Answer 1

The purpose of testing is to reduce risk. You have to decide what sort of risk you are attempting to mitigate.

I would suggest that the risk of code flaw should have already been mitigated through functional testing in a QA environment. So you don't need to exercise all of the code in your automated tests. Code coverage is not at issue.

Instead, I would suggest running a limited set of tests, covering only those pieces of functionality that are sensitive to environment-specific variables: system configuration, database availability, permissions, stored credentials, certificates, network connectivity, disk space, and the like.

It is also helpful to designate a test user or users with an easily identifiable name. That way, when your tests generate "garbage data" on reports, people can quickly see that it is their old friend "Mr. IgnoreMe." If the garbage data is likely to be viewable at an executive level, you might consider a more business-politically-correct name, like "Dr. Quality" (notice the title makes it gender neutral).

You can also restrict testing to certain time windows, outside of peak hours, so that garbage data can be identified by the time (e.g. everyone knows to ignore anything that happens in the logs between 3:30 am and 3:45 am).

Having that name out there is not necessarily a negative, as it reminds the organization of the importance of quality assurance and actually advertises to everyone how hard you are working (Dr. Quality sure is up late, and s/he's always there!)

That being said, there are times when garbage data is not acceptable. For example, with certain financial applications, setting up a fake user in a production system is actually a form of fraud. In those cases I would suggest finding an actual task performed by a real user and automating it, and putting up monitoring around that automated task. So for example if an admin signs on and prints out a set of reports once a day, perhaps automate the task of retrieving them, and send out a notification if it fails. That will at least test your authentication systems and notify you for a hard down outage. And if you are clever about it you may be able to cover more specific system aspects as well. Anything that generates durable data that is under audit control (e.g. a financial transaction) should be vetted with your compliance officer, if applicable.

Answer 2

Run read only test in production. Surely everything you do doesn’t result in a write to the database (except maybe for activity logging).

The most relevant smoke test for production is it up and responding, not whether it can do task x or y. I would say the limit of bogus data you should have in production is a login, and if your login requires additional information other than username/password (or oauth token if you’re lucky), you should (a) not even have that, and (b) rethink your design so that the minimum information needed to have a login is username/password/auth-token.

Once you start getting into can it do x or y, you’re really on a slippery slope, and whether you need to create fake data should be your criteria for having gone down the slope and fallen off the cliff at the end of it.