Selenium and non technical team members

Question

I work on a team developing a web site with a large number of pages. The QA guy on the team currently runs all of the tests manually. I am thinking of having him create a bunch of Selenium scripts. I don't think he can automate all of the tests, but he could automate many of them.

Because he doesn't have a programming background, how hard will it be for him to create and manage a batch of Selenium scripts? What issues will he need help with?

Answer 1

I think Selenium as a stand-alone "record and replay" tool is ideal for non-technical users. The idiom of recording my path and then replaying it is intuitive, and the interface, while obviously geared toward non-newbs, isn't too intimidating.

Selenium 2 / Selenium RC is obviously a higher order of complexity, and I don't think you can ask a non-programmer to interface with that. (I've been frustrated with the lack of full interface for non-Java languages, actually. I know the current release is transitional, but I really could use a full PHP solution right now, not in some mythical future release. But that's not what you're asking.)

I think if your tester guy doesn't already know about Selenium, he'll kiss you square on the mouth for telling him about it.

Answer 2

Selenium-IDE is designed for the purpose of being used by non-programmers. But my experience is that the scripts that are produced that way (using generated XPath) are very brittle.

So create in them should be no problem at all. Managing will be not-so-bad if your application is fairly stable. Or nearly impossible if your application is in development and/or changing frequently.

Selenium-RC is purely a developer tool.

Answer 3

I strongly recommend you ask this question on the Software Quality Assurance & Testing beta site, or search for similar questions that have already been asked there. You will find a much greater depth of experience with using Selenium for record-and-replay QA.

Eric's answer is the experience of most QA Engineers who used record-and-playback tools: The resulting tests were brittle and difficult to maintain. A developer can use Page Objects with Selenium RC to make a far more maintainable UI test suite, and is a better option when it is available.

This isn't to say that Selenium IDE won't be useful for your tester; however, the usefulness (and the potential to hurt testing efforts rather than help) strongly depends on how fixed the UI is. You will get your best results with record-and-playback if the UI can be frozen early in the development cycle. You may need to teach the tester how to get new XPaths when the old XPaths are broken, and how to tell if the test is failing due to a UI change (because the XPath is no longer valid) or because of a functionality bug.

I do think using Selenium IDE is better than no automation at all for any project that has some generally-stable UI. Running hundreds of manual tests on stable UI every time that there is a minor change is a real time-sink. Just set expectations accordingly and realize that changing the UI will have a significantly bigger impact on the QA schedule once you start using record-and-replay automation for that UI. Your tester may want to pick-and-choose the areas s/he automates wisely.

Answer 4

I've had no problems training non-programming folk to write Selenium tests with relative XPaths in Java/JUnit. The programmer sets it up and writes a basic test template, the tester develops more tests - by recording and then tweaking them. If tester needs anything specific, he asks programmer to write it. Tester needs to have a really basic understanding of HTML and XPath, and a bit of help to pick up Selenium API (as well as Selenium plugin for Firefox and Firebug).

The funny thing is, at first I was considering something fancy like Cucumber or something similar, but this Selenium/JUnit combination proved good enough and simple enough. Curiously, testers without previous Java experience had no serious problems writing JUnit tests. The only important thing is that experienced programmer sets up the test, so that the tester basically just needs to produce new tests and call Selenium methods.

Answer 5

Consider using a keyword-driven testing framework. The idea is, developers develop high level keywords (or you use the ones that come in a library), and the tester simply assembles these keywords into high level test cases.

The teams I've been on over the past few years have gotten a lot of mileage out of this approach. We use robot framework but there are others such as cucumber (supported by several languages) and specflow (a cucumber implementation for .net)

For example, using this approach your tester might write a test that looks like this:

| Login with valid credentials
| | Go to page | LoginPage
| | Login as a normal user
| | The current page should be | DashboardPage

These keywords could be written in python (or java, or a .NET language), like so:

class LoginPage(PageObject):
    def login_as_a_normal_user(self):
        username = self.browser.find_element_by_id("username")
        username.send_keys(DEFAULT_USER)
        password = self.browser.find_element_by_id("password")
        password.send_keys(DEFAULT_PASSWORD)
        ... and so on...

There is a pre-made library for robot with generic keywords based on selenium ("go to page", "page should contain", "click button", etc) with which the tester can immediately become productive with.

Quite often these generic keywords are sufficient, but using the concept of page objects and page-specific keywords is powerful way to write tests. Your developers can create a class for every page in your app, and for each page they can write special-purpose keywords so that the tester can focus more on the logical function of the page rather than the physical implementation.

Here's a couple blog posts I wrote that dives into more detail in using page objects with robot framework:

• Robot Framework Page Objects
• Using Mixins with Page Objects

Answer 6

Actual experience from non-technical team members writing Selenium code: it didn't go well

We actually had this very situation happen where we had a non-technical team member (in this case, an SQA with no programming background). This unfortunately didn't go very well.

Record and play created unmaintainable tests

First our team had tried the record and play tools, but as other have said, the tests it generates are very brittle and hard to maintain. Our SQA eventually wound up falling into a pattern of just rerecording a test every time it changed, which wasn't really that efficient, especially when one change broke most of our tests (we had one change to our website's main page that broke about 60% of our tests).

Without help from those with a programming background, the manually written code was horrendous

We wrote our Selenium tests in Java and the SQA had never used Java before, so he was learning it on his own. We found that tests wound up using some really poor programming practices:

• Using public static variables when they really should have been private instance variables
• Having code blocks that didn't do anything
• Lots of Thread.sleep() instead of using WebDriverWait because he couldn't figure out how to write custom conditions
• Really awkward unit testing: I saw quite a few assertTrue(false) lines to terminate a test
• Poor variable names
• Suppressing exceptions that should have been handled (or prevented from occurring in the first place)
• Tests that passed no matter what input you used
• Some code we never did figure out and we just wound up rewriting

When I arrived on the team, the original SQA had left and running any test resulted in a bunch of console exceptions that we were just told to ignore. After that, we wound up getting developers involved and massively rewriting a lot of the code to make it actually work correctly and be maintainable and easy to understand.

If you're going to have non-technical people write Selenium code, have a technical person help them

I think that some of these problems might have been avoided if we had someone who was technical come and help them. If they had explained why public static variables should instead be private instance variables, or how JUnit works, or how to use WebDriverWait, or why scattering Thread.sleep() around everywhere is bad, we might have had better code.

But as is, we wound up with code that was ultimately unmaintainable and in the end, we just rewrote most of it, resulting in a big waste of time and money.