0

Is there an industry standard in how many times a scenario is tested before it is considered passed? Is it common to test a given scenario using multiple different records?

Here is the issue: A development project is now in the SIT phase and the developers' plan is to test each scenario once unless it fails. There is an opinion that the scenario needs to be tested more than once before considered passed because, in the past, there were defects found during UAT for scenarios that passed in SIT.

  • Do you know why the scenarios previously passed SIT but failed UAT (poor testing, misunderstood requirements, insufficient coverage, ...)? Have you made any changes to your testing processes to account for similar issues? How are you defining a "scenario" here? – jonrsharpe Jul 27 '15 at 14:11
1

IMHO you are asking the wrong question. Does it really make a difference for your software's quality or the efficiency of your tests if there is an "industry standard" or not? There is no widely accepted standard, nevertheless what you should ask yourself is: "does adding multiple different records for the same test scenario improve our process?"

My suggestion is to approach this from a pragmatic point of view. Test cases are good if they have a potential to reveal defects. If your current set of test cases has overlooked a defect which was found later, it is obviously a good idea to add one or more test cases to the pool which can reveal this defect (or even better: defects of the same type) to let this not happen again. If these test cases count as "the same scenario" or "a different scenario" is pretty much irrelevant for you and your team - if they are useful, add them, if they are only testing things already included in other test cases, omit them.

And do yourself a favor and stop thinking in terms of "lets do this because everyone else does it" - that is an organizational anti-pattern called cargo cult programming (it's description will fit for "testing" as well).

0

The next step beyond 'testing every feature once' is systematic Pairwise Testing. This typically results in a set of a few tens of tests per feature, depending on the features visible complexity. This is obviously a lot more work, although not necessarily tens of times more work, at least if tests are automated. You will almost certainly need a suitable tool.

Alternatively, you could bake in robustness at a lower level, with developer-level unit or integration tests. At that level, instead of writing out tests cases one by one you can use a tool like Quickcheck and/or JUnit theories. Links go to java tools; most languages have something equivalent.

Whether you do either, both or none of those would depend on the nature of the faults you are seeing at UAT, and how much of a priority it really is to reduce or eliminate them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.