In an ideal world, dev/ops are working together as one team to release something onto production.

However, there are organizations with limitations so that release engineers/teams are kept completely away from development/project teams.

This not only added the communication pain for releases but also making the production environment almost "invisible" to project team. To some extent, there is "trust crisis" between project and deployment team.

Even we have good and reliable Acceptance/Functional tests in UAT environment but we just not sure if the deployment process has been done properly and if everything works on production exactly as UAT. So the only way we can verify this is through some tests on production environment.

Some simple smoke tests could be helpful to identify connectivity problems.

So my question is : What's your opinion on applying all tests we had on UAT to production (with configuration to disable the bits that we don't want to happen on production such as removing data etc.)

If you have done something similar, how do you play data on production environment without having audit trails tainted (one obvious thing would be account based, but let's assume there isn't segregation by account).

EDIT: I didn't mean to test prod environment all the time, but only right after deployment. Also we will not stop trying solving the problem from the root - dev/op relationship. But at the same time would like to explore other possibilities.

closed as too broad by gnat, GlenH7, Tulains Córdova, david.pfx, user40980 Aug 8 '14 at 1:03

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    What sort of failures are occurring on the production side that didn't get caught on the UAT side? – Robert Harvey Aug 6 '14 at 22:23
  • "applying all tests we had on UAT to production" -- wonder how that would work like in a project where UAT testing involves things like completely cleaning up database or like artificially creating conditions like date moving a year or two forward, necessary to verify particular usage cases – gnat Aug 6 '14 at 22:35
  • @RobertHarvey they are varied but mostly due to the deployment process being slow and bureaucratic hence error prone. We have the "hand over" culture so that once it is on deployment engineers we don't have a way to verify if the deployment is done correctly or not. – user2001850 Aug 7 '14 at 7:43
  • @gnat just like I mentioned, we could have some configuration on tests to run them in UAT DEV or PROD. We could play in some dummy data but never delete them (on PROD) - however this will be part of audit trail. – user2001850 Aug 7 '14 at 7:45
  • If one of your concerns is improper deployment/installation then you should have an application that automatically performs the installation and does some validation checking to ensure the installation was successfully installed or not. Verifying that the software runs the same in the production environment versus the development environment is another matter. While not perfect, a duplicate production setup just for the development team is ideal. But even there, the real production system will have its differences. – Dunk Aug 7 '14 at 16:16

Usually, tests are performed outside production servers for three reasons:

  • Safety. If something goes wrong and all data from staging is accidentally removed, you don't care. If it happens in production, it's... let's say annoying.

  • Performance. An extensive suite of tests can easily use 100% CPU on several servers for several minutes. You can't afford slowing down production servers for a few minutes ten times per hour.

  • Goal. The goal of testing is to prevent issues in production. If you find an issue when the application is already deployed, it's too late. Moreover, reverting to the previous revision can be complicated. For example, what if the new version created a column in the database, and meanwhile, you have new records where this new column was used?

For those reasons, I would strongly advise you against the usage of production environment for testing. Moreover, I don't think running tests in production and sending the results back to your team will help to gain the trust of system administrators; they may perceive that as a tentative to circumvent the limitations they set to protect their infrastructure from your team.

Instead, address the original issue, i.e. the lack of trust between developers and system administrators. Until communication and trust issues exist on this level, the testing and deployment process will  be painful, no matter what tricks you'll use to make it easier.

  • I agree and we will be continuing trying to solve the problem from the root. But at the same time we'd like to explore other ways to help us making the process less painful and error-prone. – user2001850 Aug 7 '14 at 7:46

Provided you've implemented a robust version control system you spread the tests out so that you just need to confirm basic functionality.

As an example the following are done in the four environments (simplified for brevity):

  • DEV: Unit tests, some behavioural tests
  • FAT: Integration tests, regression tests, installation tests
  • UAT: Test of deployment to LIVE (possibly with faux data)
  • LIVE: Common functionality, possible environment testing

DEV (Development) environment

The focus is on unit tests to test the new functionality, bug fixes, and the previous tests; some behavioural tests can be performed if required for new/modified functionality.

FAT (Factory Acceptance Test) environment

You test your installers and allow your testers to run through the application and test the new functions, bug fixes, and to ensure the old functionality isn't broken through regression testing.

UAT (User Acceptance Test) environment

Your client/customers/users test your application to ensure it works according to their expectations. Of note is that UAT should be identical to LIVE so what works here should work in LIVE.

LIVE (Production) environment

You just need to get the update to the latest version done, and recheck basics (smoke-tests): can access data, open reports, view documents, export/import data, etc.

By the time software gets to the LIVE environment is it has passed all the previous environments without issue, so there is no need to retest; if any environment fails for any reason the software release goes back to DEV, changes are made, and the whole process starts again under a new release number e.g. 1.2.3.0 -> 1.2.3.1.

You build trust in your release to LIVE by going through each environment and completing tests so that you don't need to retest in LIVE.

  • Eeee. If deployment team is kept separate, development team have no way to assure that UAT/FAT/LIVE share the same environment. Thus there will be bugs due to mismatch. – przemo_li Apr 5 at 7:41
  • In my scenario above only UAT and LIVE would be the similar, but in separate environments, as each environment has a specific purpose. It would be the deployment teams responsibility to ensure UAT and LIVE "match", by controlling the changes that get applied and verifying the changes are as expected. Mismatches would typically be an infrastructure issue (e.g. network, permissions, lack of resources) or a deployment mistake. Verification is vital. – Kevin Hogg Apr 9 at 12:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.