At least few times a week I have to track down a bug that would not occur in a production\client environment. 95% of the time I can tell that its a QA configuration\environment problem just by looking at the ticket. But, many times Im asked to track down and prove that its a QA issue and not an actual bug.

How common is this?

If QA doesnt know when there is a legitimate bug then I think there is a big problem.

I've suggested setting up the QA environment myself to avoid these problems.

Who should setup the QA environment? Developers or QA?

  • 1
    In an ideal world the creation of your Production, Staging, and QA environments would be scripted. – MetaFight Sep 15 '15 at 15:24
  • urg yes v common – Ewan Sep 15 '15 at 15:46

All the companies I have worked at (and I know I am lucky because of this) have had QA that is set up exactly the same as production so I am unsure how common those sorts of bugs are.

Deployments to QA were completed the same way deployments to prod were. If there were deployment scripts the only thing that changed between UAT and PROD were server names.

I would say that bugs in QA are actual bugs with the software and are legitimate. A bug resulting from a configuration error is still a bug (even though a configuration change can fix it). I would keep track of all the bugs that are a result of configuration differences between QA and PROD and then ask management for funds (if needed) or authorization to make changes so that the configuration between QA and PROD is the same. If the environment is set up for every QA test work on the scripts so that it is the same as prod.

By having our QA and PROD set up the same we could try out new configuration changes in QA and see if it would introduce any bugs or cause performance issues. With out QA and PROD matching that would not have been possible.

As for who should set up the QA environment? It has varied from the devops team (responsible for all environments) to the server admins (responsible for all environments). However this will differ for every company.

Do not question the "legitimacy" of bugs. If some one finds a "bug" it means something did not work correctly. If it is because of a configuration issue either fix the configuration issue in QA or handle the issue in code if you expect that configuration issue to be present in PROD. As soon as you start to question the "legitimacy" of bugs things will get missed by QA and some major catastrophy will happen in PROD, Murphy likes to rear his head.

  • I agree with you for the most part. Issues found during testing should be treated as such, as long as you know QA is the same as production. But in a QA environment QA techs have unlimited access to the system. They can, and do, hack at the database, file system, or host of other system dependencies that would not be touched by end users directly. Therefore, there is chance that the system didnt start just like production. Then, someone must spend time figuring out what happened because well there is an issue, but is there? This is an ongoing time consuming thing. – andrewramka Sep 15 '15 at 16:32
  • @andrewramka I would never have an IF (QA) type check in my code. I have had situations where an expected configuration value in a database table could validly not be present in PROD and the code would handle that. If the situation would never occur in PROD or was required the code would fail in QA. – RubberChickenLeader Sep 15 '15 at 16:39
  • @andrewramka Why do QA have unlimited access into your system? I think something is terribly broken in your environment management at this company. Don't mistake this for being normal. QA should under all circumstances be a mirror of production. – maple_shaft Sep 15 '15 at 16:53
  • Ideally the only difference between test and production is the most recent software build being tested (slight staleness of the data is acceptable, like weekly refreshes of test from prod). Configuration is part of the deployment, so if that broke, it should be considered a failure of the deployment of the current version. – cdkMoose Sep 15 '15 at 18:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.