Is it unreasonable to implement a small domain specific scripting language?

Question

At my new job, they are currently spending scores of labor hours that we don't have to do manual QA testing after every build. Nothing is automated at all. (We can only afford three developers, and they are all far too busy to be involved on the testing side)

So I'm building an automated testing framework that basically models the system and allows a Selenium driver or similar to make manipulations on the real web app based on the in-memory model, using a semantic interface. I'm pretty pleased with it. But every test case is currently a code change and I'm the only one with both the ability and the time to do so.

Would it be unreasonable for me to create a small, simple scripting language to allow less technical coworkers to write tests? Is there a better way to do this? I want, as much as possible, for coworkers to write business requirements in plain English that my program then tests our app to ensure it meets the requirements.

If it's a reasonable solution, but seems like overkill for this specific project, that's fine; it's also a portfolio builder.

Answer 1

I think it's reasonable to use a small DSL, I'm not so sure it's wise to build one considering you don't have much bandwidth. There are frameworks that exist that can do what you want.

For example, robot framework allows you to write your test in high level keywords. As a test engineer you can write these keywords in python, java, .net, or other languages (and there are pre-built libraries for all sorts of things like testing web apps with selenium, web services with REST or SOAP interface, desktop apps, etc). Your less technical coworkers can then assemble these keywords into tests.

This particular framework lets you write tests in a BDD style (Given/When/Then), a data driven style (tables of inputs and outputs), and a procedural style, which means you're not locked into one specific style for all of your tests. In my experience, different scenarios more naturally lend themselves to one style or another, so being able to be flexible gives you a lot of power to express yourself appropriately.

Robot framework tests are plain text and mostly read and write like plain english. The more effort you put into building a catalog of keywords, the more english-like it will be.

For example, here are what some test cases for a simple web app might look like:

*** Setup ***
Library  PageObjectLibrary
Suite Setup
...  Restart the server
...  Open browser  ${app_root}
Suite Teardown
...  Close all browser windows
...  Shut down the server

*** Test cases ***
Valid user can log in
    [Setup]  Go to the page  LoginPage
    Enter valid user credentials
    Click the login button
    Current page should be  DashboardPage

Invalid user goes to the "invalid attempt" page
    [Setup]  Go to the page  LoginPage
    Enter invalid user credentials
    Click the login button
    Current page should be  InvalidAttemptPage

Dashboard of new user contains no projects
    [Setup]  Log in to the application as a new user
    Verify that the page contains a list of projects
    Verify that the list of projects has  0  projects

There are other frameworks you can choose from that work in a similar fashion (fitnesse, cucumber, . The choice depends on type type of application your testing, and how technical your testers are.

Granted, with any framework you will need to devote time to learning the framework, organizing your tests and so on. In most cases, however, that time is much less than you would spend writing your own framework from scratch.