6

We have a software product has a lot of usability features, and we're constantly changing it. We have issues with things like persisting scroll position which seem to get fixed, but then are broken again when we release. We have good testers, but we seem to miss things like this because there are just so many features in the application it's hard to perform thorough regression testing.

How is this problem best solved? I'd love to have automated UI tests, but those are fragile. I'd love to have our testers test every feature every time we implement a new feature, but that would take forever, and seems impractical. I wanted to gauge how other software shops do this and how they avoid recurring bugs.

Improve this question
1
  • 3
    You might get better answers at sqa.stackexchange.com, since building maintainable automated UI tests / deciding when to automate is really a tester "thing"
    – Ethel Evans
    Jun 15, 2011 at 22:47

4 Answers 4

Reset to default
11

Unfortunately for you, that's exactly how other shops (the successful ones, at least) avoid recurring bugs. You create a suite of regression tests, either automated or manual, that cover all of the bugs that you've fixed (or as many of them as you can, at least), and then you make sure that the whole set is run and passes before letting a build see the light of day. If any of them fail, the build fails, and you go back and fix the problem, the re-run the entire test suite. It takes discipline, but it will greatly increase the quality of your releases.

Having good source control and designing your code to make it easier to test really helps simplify this process. Also, I'd recommend reading up on good processes for build management. Release Management Done Right makes a pretty good starting point.

Improve this answer
1
  • 1
    +1: Regression testing helps in two ways. The less common one is where a bug fix vanishes out of the code, like when somebody does a copy-and-paste and obliterates the fix. The more common one is, well, some chunks of code are just fragile, and if there was a bug there once, a regression test on it may find another bug there later. But there are no shortcuts, especially with GUIs: somebody has to sit down and run through the regression tests, or you'll just keep seeing bugs reappearing in your releases.
    – Bob Murphy
    Jun 15, 2011 at 22:49
3

Automate below the UI layer when possible to catch your functionality bugs. Develop by keeping your UI layer as thin as possible. The MVC / MVP patterns might help with this. Whenever possible, test the class library or API underlying your UI instead of the UI itself.

However, when it comes to UI bugs like the scroll bug you mentioned, you really do need to write some UI tests. There are things you can do to minimize the effort, however:

  • Use IDs for all of your UI elements when developing the product, and record each ID in a single variable in the test code. Use that variable everywhere else in the test code when you need to interact with the UI element. This way, you don't have to update multiple lines of code. In WatiN, you can write classes that inherit from the Page object and use the FindBy attribute to list their ID just once in your code.
  • Firm up the UI early in the development process. Stylesheets or graphics can change, but your basic control and UI elements should be set in stone ASAP. The sooner test gets those IDs and user stories for using the website and knows they can rely on them, the sooner they can start writing maintainable tests. This might require process changes.
  • Automate only those tests that cover areas that are not in constant flux. Go for the low-hanging fruit on UI tests. If you aren't sure, keep running it manually for a while longer.
  • Automate especially those things that "never" break. These are the things that testers are most likely to miss (because they aren't likely to have problems) and the things that are most maintainable to automate. This will free up testers from worrying about relatively stable areas of the code so they can focus on those difficult-to-automate areas that are more likely to have bugs.
Improve this answer
1
  • For anyone else who's curious, "WatiN" is "Web Application Testing in .Net", and appears to be one of those acronyms that adherents like to throw around without ever expanding, on the assumption that it should be obvious to everyone what they're talking about. (I say this because I had to follow three links before I finally found a page that did expand it.) Seems oddly specific considering the original question never mentions .Net, or even Windows at all.
    – FeRD
    Mar 29, 2020 at 20:52
2

Use automated testing for everything you can. If you feel that automated UI testing is fragile, that doesn't mean you can't use automated tests anywhere in the program. As the program grows, the tests do too and you will have a stronger handle on what is happening and that it is happening correctly. Anything that doesn't pass automated testing doesn't get tested by testers. This gives them more time to test the harder to automate things, as simple regressions are caught early.

Improve this answer
2

The only thing that's going to come close to finding this sort of problem early is unit testing.

In this case you'd have a set of unit tests that concentrated on this problem area and you'd make sure that these tests were run before checking in any change. Then if the tests failed you'd know that the edits needed more work.

As the problem is in the UI that does make testing more difficult, but you have to ask yourself (and the boss) - what's more costly, getting a set of unit tests working or having to keep revisiting this code again and again and still not being sure that you've got a stable solution.

In the absence of unit tests you just have to run the tests manually and make sure that they're run early and often.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.