How can you write tests for Selenium (or similar) which don't fail because of minor or cosmetic changes?

Question

I've been spending the last week or so learning selenium and building a series of web tests for a website we're about to launch. it's been great to learn, and I've picked up some xpath and css location techniques.

the problem for me though, is seeing little changes break the tests - any change to a div, an id, or some autoid number that helps identify widgets breaks any number of tests - it just seems to be very brittle.

so have you written selenium (or other similar) tests, and how do you deal with the brittle nature of the tests (or how do you stop them being brittle), and what sort of tests do you use selenium for?

Answer 1

The purpose of Selenium is to create UI-driven integration tests.

Integration tests verify that all components of your system work correctly when deployed together. Integration tests are not a sufficient test strategy and complement other test strategies having a different focus, for example unit-testing and acceptance testing.

UI-driven tests are inherently brittle, although Selenium and Watir are a step up from the early days of record-and-playback tools. There are several ways to deal with this problem - here's is a compilation of advice from some world-class experts:

Don't try to get all your test coverage from this type of tests. Robert C. Martin argues that your code coverage by integration tests should be about 20%. It is simply impractical to test all paths of execution when the input is several application layers away.

Get most of the test coverage from unit- and acceptance tests. Look for links to Gojko Adzic's articles in FinnNk's answer. Adzic argued repeatedly about testing business logic through acceptance tests and bypassing UI.

But some amount of UI-driven tests still needs to be written. This is where you need some practical advice in addition to "don't test your business logic via UI." I'd recommend Patrick Wilson-Welsh's blog as the starting point.

Answer 2

The most important thing when creating tests like this once you get past a trivial number is the idea of symmetric change - a small change in the code should result in a small change in the test suite.

For example, let's say you collect someone's name with two text boxes in 100 tests. If you write those tests niavely (or maybe are using record-playback) then you will have 100 tests to change. If instead you abstract out an 'enter name' step and use that in your tests instead of directly using selenium then you only have 1 place to make your change. It'll depend on your context how many layers of abstraction you use - but using abstraction will go a long way to making your tests maintainable.

A second important thing is to make sure your selectors are based on the thing which is most important and least likely to change. Sometimes this will be an id or a class or it could be the text in or surrounding an element. Always prefer the simplest selector (e.g. an id) but sometimes to achieve this you will have to use something like xpath.

Gojko Adzic has lots of great advice on his blog when it comes to this sort of testing - check out his Sine of Death and how to avoid it in particular.

Answer 3

Testing ensures that code behaves as expected. If your tests break on a regular basis, then the tests are either not testing the right thing or the developers do not care about the tests.

Personally I think that if the presence of a <div> tag breaks a test then the test is needlessly strictly dependent on the surrounding tags, and a more lenient approach should be taken.