How much detail is in a good UI regression test?

Question

We use a detailed step-by-step user-interface regression test for our commercial web application. It has a "backbone" test for the most used / most important parts of the system, with optional tests for specific areas of functionality. Using this plan has definitely helped us ensure high quality software.

But, having very specific tests can be counter-productive. The tester concentrates on following the test and will completely miss usability issues, or not notice fairly obvious problems such as the bottom part of a page that is missing.

By contrast, some of the best UI testing happens when building a demo of a new feature. I often do my own best testing by pretending to demonstrate the system to an imaginary prospect. Yet when I tell the testers, "Just demonstrate the system to yourself" they don't cover nearly as much functionality as they do with a detailed point-by-point test.

I'm repeatedly asked to provide more and more detail in the test plan so that a new untrained tester can test with it without asking any questions. Yet details seem to be counter-productive. How much detail do you put in a regression test to make it effective? How do you write tests that make the tester focus more on the system than on checking off items on the test?

Answer 1

Technically speaking, a regression test should not be looking for usability issues. It may catch deviations from the UI design as part of the test cases but usability issues are best determined by UI design specialists. UI regression tests are usually performed by QA (testers), as you know.

If the product is too big for a detailed UI design, the UI designer produce should produce guidelines that the development team should follow and the QA should check for. For example, "Each page should contain a header with A, B and C" or "When asking for an address, it should have these fields laid out in this way".

It may be better to supply the UI verification steps in a different way. For example, instead of giving steps, include screen shots of that the app should look like (which should catch the bottom portion of a page missing).

You may also want to consider an automated UI testing tool like Selenium. This can automate much of the repetitive testing aspect you are capturing in the steps and free you and your testers up to look for harder to reproduce or more critical issues.

Answer 2

You cannot command someone to be innovative. If you have only testers in your team without developing own ideas for good tests, I don't think a better test plan will solve that problem for you. Either they need more experience and training, or you need different people in your team.

Said that, I think you should have both:

1. a very detailed step-by-step check list of things where you want testers to have an eye on

2. some general questions / guidelines, for example, about testing usability or about things not explicitly mentioned in your plan, or a demand for creating missing points in the checklist by themselves.

Make clear that you expect your team not only to focus on point 1, but also on point 2, but don't expect too much because of what I wrote at the beginning.

Answer 3

If you say "demonstrate the system", I hear customer workflows: how the customer is going to use the system.

So typically, next to some basic functionality, like "does it install, start etc.", I find UI testing mostly helpful to simulate a real user working with the system.

Also, most often, UI automaters get their specification from some other place. If there is no such place/person, you could ask them to talk to customers or product management on how they use the system.

This also has the benefit of you being a little more certain that the product you ship is working in (at least) some real world scenarios.