7

I'm looking for guidance on how to keep my acceptance tests isolated. Right now the issue I'm having with being able to run the tests in parallel is the database records that are manipulated in the tests. I've written helpers that take care of doing inserts and deletes before tests are executed, to make sure the state is correct. But now I can't run them in parallel against the same database without uniquely generating the test data fields for each test. For example.

Testing creating a row i'll delete everything where column A = foo and column B = bar Then I'll navigate through the UI in the test and create a record with column A = foo and column B = bar.

Testing that a duplicate row is not allowed to be created. I'll insert a row with column A = foo and column B = bar and then use the UI to try and do the exact same thing. This will display an error message in the UI as expected.

These tests work perfectly when ran separately and serially. But I can't run them at the same time for fear that one will create or delete a record the other is expecting. Any tips on how to structure them better so they can be run in parallel?

2 Answers 2

10

It personally sounds as if your Acceptance tests have encompassed properties of Integration tests and that you are trying to "kill two birds with one stone" as the saying goes.

In the traditional Waterfall model a single Acceptance test should determine if a single requirement has been met. If developing based on a strict SRS document, you may find that even basic input validation is explicitly defined and by the nature of Acceptance testing it needs to be manually tested to be verified.

In the Agile model however the Acceptance Tests verify a single user story, a high level test to verify a user high level stakeholder business need. Typically in the Agile model such fine grained control and specification over concerns like input validation should be understood, unless that validation is unique or specific to a business need.

Simply put in any case, in the example where you wish to verify a duplicate record is not entered into a database is far too low level for a user story, and one would argue is a waste of valuable time to elevate to the importance of an Acceptance Test. Quality assurance or the tester for that feature should be able to verify that the high level requirement has been met with no obvious defects.

Your tests need to be split up:

Automated Unit Tests

For your lowest level tests, typically written and performed by the developer to verify all of the functionality of a specific component or application layer apart, independent of other areas of the application, and reproducible to run multiple times.

Integration Tests

These tests like unit tests can verify situations like creating a new Person record, across all system layers, verifying the integration of all application dependencies while verifying that creation a new Person record, or preventing a duplicate Person record from being created is occurring correctly.

The Case for Integration Tests

One of the most valuable aspects of these kinds of tests is that their are various strategies to not only Automate these tests, one can also use database transactions to make them run in parallel and not conflict, and also this database transaction can be Rolled Back when the test is over reverting your database back to a clean slate and making them reproducible.

I gather that based on your question that most of your "Acceptance Tests" aren't terribly interesting and would be better automated. This isn't to say that Acceptance testing shouldn't occur, but it should be done on a much higher level to where these issues you bring up no longer matter.

8
  • So what happens if one of the requirements IS that duplicates aren't allowed (one record in the database actually maps to a single entity in the domain). Nov 6, 2012 at 14:21
  • It's times like this when the internet sucks, because I can't actually show you what my tests look like. But if it helps I'm not really doing an integration test. I'm not looking at the database at all. I'm using selenium and specflow to actually manipulate my website in code. Nov 6, 2012 at 14:24
  • @MattPhillips Your issue is two fold. Your requirements should not call out database records and common knowledge. If a user doesn't care/know about it then it shouldn't be a requirement. Secondly, Selenium is a great tool for testing the application UI and presentation logic, but this is no substitute for a true acceptance test. An acceptance test should not concern itself with duplicate database records, just that I can't create a user with the same ID. This kind of test is too low level for the UI to do well as you are experiencing. cont...
    – maple_shaft
    Nov 6, 2012 at 15:22
  • ... Try writing repeatable unit tests that use transactions for testing low level functionality and leave your Selenium tests for the more interesting tests involving UI interaction. Spring has really good support for transactional JUnit tests that automatically rollback. It is worth looking into.
    – maple_shaft
    Nov 6, 2012 at 15:24
  • Ok, so your example "just that I can't create a user with the same ID" doesn't map to my domain though. My test shows that when you try and create something with duplicate data, an error message is shown in the UI, that to me would be the acceptance piece. But how do I structure it so I don't end up with 10's if not hundreds of fake data records just to use the UI properly. My application is very data driven. So i can't just fill the fields with garbage in order to test. Nov 6, 2012 at 15:27
1

At Intelliware, we use "home state switching" to address this problem. We have an internally developed framework called Runway that abstracts away the complexity, but here's how it works in a nutshell:

  1. When a tests starts, it makes a remote call to a home state controller get a database to use for the test.
    1. The home state controller creates a new database from scripts, or uses one from the pool of available homestates (created, but not currently in use)
    2. The unique UUID is returned to the test
  2. The test includes the UUID as a header on all requests
  3. The system-under-test detects the UUID header and routes all database calls to the test db. This is how you get parallelization - different requests get routed to different databases depending on the header.
  4. At its completion, the test returns the database to the 'available pool'.

This approach depends on being able to spin up new database schemas quickly and reliably. We typically use Flyway for this, but eventually you may need to export a snapshot if the migration starts taking a long time.

The aforementioned home state controller is a standalone, lightweight web application that never gets deployed outside a test environment.

See also

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.