Our test automation team writes tons of automation tests for our WEB application, but it seems to me that their code is becoming way too complex and warrants that someone will test it as well.

Is there a best practice for writing these automation tests? I'll appreciate any answer with concrete inputs from someone who experienced this problem.

If it matters - we are using jSystem for automating our WEB application tests - if there are frameworks which simplify life - I'd like to hear about them.


You should give an example of what "too complex" is to you.

In general, tests should exercise a small part of a system while keeping everything else constant. For example, if your website reads documents from a database, you can test the web site against a prepared database of known documents. Having said that, testing features on websites is hard because websites are large and complex systems, while your features may be fairly simple.

If your tests require a fully running system, then they are truly "integration tests", not "unit tests". To make sure that the page "/user/RonK/profile" shows "ronkitay.wordpress.com", you will implicitly also test that this page returns an HTTP status code 200, and that the response body is text/html. If your tests are running inside a browser, then you'll also be testing that browser's handling of the HTML DOM and any client-side scripts. Finally, you'll test that "ronkitay.wordpress.com" is on the page. If it isn't, then any of the previously implicit conditions may have failed -- the component that renders RonK's website URL may be coded correctly, but simply hasn't executed. A tool like Selenium will let you control an instance of a Firefox browser and test expectations/behaviors by inspecting the DOM. This is usually okay, but it doesn't test Safari or Chrome or IE.

One thing that's often overlooked in automated website testing is the page layout -- the DOM may be correctly organized so the things you are testing for will exist, but they may be laid out very poorly (or not at all, if the CSS file isn't referenced properly.) This is overlooked because it's hard to test layout conditions: how do you express this in code: "the Community Bulletin page element must not spill over to the Related element, the text must wrap nicely, and links shouldn't be underlined"?

What I've seen is that parts of the website content generation can be isolated and tested without the web server. If you use an MVC framework (like SpringMVC), then models and views and controllers can be tested independently as pure Java code. (The jUnit framework will let you do generic tests pretty easily.)

By testing the M, V and C parts of an MVC-structured website, you focus the testing on the business rules in your code, which is usually where the bugs will be. For example, you can build (or mock) a fake user object that is given to the UserProfile model. You will expect that this UserProfile model's websiteURL field will be correctly set. You can then couple the UserProfile model with its view (the SpringMVC framework does this normally, but you can do this yourself as well) and confirm that the result is correct HTML for that portion of the page.

As for your comment about testing tests ... if there is complete coverage of your code, then the tests have exercised everything there is and you are probably okay. (They may be difficult to extend, or modify, but those are design choices.) If that isn't enough, you could have two people independently write tests for the same thing and look for consensus. ;-)

  • It's hard for me to explain what I see complex in the tests - it is mostly a "gut" feeling. Our tests our true integration tests, in your experience, how much integration tests would you "trade" for unit tests? Would you use Unit tests for 80% coverage and use integration for the 20% remaining? – RonK Apr 14 '13 at 9:16
  • 2
    I don't recommend placing unit and integration tests on a single axis and making tradeoffs. Unit tests are for small well-defined units which have limited inputs and outputs. They should be as near 100% code and branch coverage as you can make them. A large system can then be composed of these reliable small units. Integration tests exercise this arrangement of units, and challenge the design of the product. I'm not sure how to compute coverage for integration tests, because you may be integrating with software that you didn't write -- for example, Tomcat or Firefox. – AnAnswer Apr 14 '13 at 9:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.