My boss had an idea of simplifying tests to English like statements so that non programmers could write tests as well. We chose Lisp as the syntax in which the tests would be written. This testing framework is built for web applications and uses Selenium.
By English like syntax I mean we've abstracted away most of the technical details away so that non programmers can write tests easily. They needn't bother with trivial stuff like logging, opening browser instances, python libraries, reports etc.
We use Python to interpret the tests written in a Lisp like syntax, get some custom output from them and then feed it to Selenium.
The tests can contains statements that do stuff or just check something. Nothing else. Say it can click a button, it can check if there are 4 input boxes etc. You cannot give any conditions.
(click (select "#searchButton"))
(expect (select ".result") "count" 0)
(expect (select "#resultcontainer") "contains" "Nothing Found")
The test writer cannot use loops, if statements and variables as we do not want a programming language. Just statements to do stuff or check stuff.
The problem we currently face is maintainability. We've written quite a few tests to insert new bills into a database. We also have tests to check search functions pertaining to those bills. Search results are expected to change which will lead to failing tests. With no access to conditionals the tester has to be exact every time.
Things I've considered:-
Version controlling the development database.
The testing framework works on Linux and Windows. We will have to write scripts that will replace the existing database with a version controlled test database and then switch it back after the tests (or configure the app to use a different database). We'll have extra work when we run migrations in the database, we'll have to migrate this database as well. All these are trivial tasks but I don't know of a way to do this without making it more complex for people who aren't programmers. Once the Database is back to the old state tests would run as intended.
Specify steps to be manually undertaken by tester in the results page.
Along with the display results I could lay out some instructions to be followed by the tester manually. Please test "x" and "y" manually. Make new bills under a random customer and delete them.... & so on. This has a downside that we're going to waste time. On the plus side technical details aren't a roadblock.
I want advice that will help me with any of the following.
- Some measures to version control and restore a DB without much effort on Linux and on Windows. We use PostgreSQL.
- Pros and cons that I am not yet aware of.
- Other ideas to improve this process.