Acceptance test execution strategies for heavy first-time-setup apps

Question

We have a system with about 250 tables. The first-time-setup include running about 50 queries (known also as seeds).

We're trying out strategies to automatize our acceptance tests, and our first choice was to clean and rebuild the database on every scenario (aka: every test), so that we could keep them isolated from the rest: meaning a "list all resources" scenario wouldn't check for a resource created in a previously ran "create" scenario.

I'm not sure, whether this "isolation" will bring any measurable benefit, but it does make each scenario execution time longer (about 20 seconds each scenario).

Is it considered a bad practice, or will it bring any known scalability issue, if we change our strategy to do this first-time-setup only at the beginning of the setup phase of the tests execution (or maybe at the beginning of each test suite) rather than on every scenario?

If it helps, we're testing using Behat (the official PHP version of Cucumber).

Answer 1

This is not by any means a bad practice. It may seem like an overkill but doing integration/system type of testing such a way gives you tremendous benefits.

Tests become a lot less fragile when you set them up this way. Imagine you add a new test only to find out that all the other tests that run afterward now fail. This tend to happen because those other tests depended on a previous state that you now changed. What do you do now?

1. You can either change the way you write your new test so it does not change the state for the consequent tests
2. You update all other tests behind the new one

Both options are pretty terrible. Option #1 constrains you too much so instead of writing a test that actually verifies the use case fully, you write a test where you only kind of testing a feature. Option #2 will take forever and leaves you with a very sour taste in your mouth.

If you apply the strategy where tests do not depend on previous state, you only have to run that single test knowing with a very high probability that it will run fine as part of the entire suite. Therefore it takes a lot less time from development to full integration. The ability to run a single test on its own makes it a lot easier to isolate and debug issues. The context which a developer needs to hold in their brain is also minimized this way making it easier to focus on the task at hand. To summarize it will take a lot less time and also a lot more fun to develop and maintain such tests.

As you mentioned one negative to setting up the state before each test is that the overall running of the suite takes a longer time. I agree this is annoying and you have to consider how often these tests need to run. In my teams acceptance type of tests only ran at most once a day but not every check in and they had their own dedicated build (in other words, they were NOT part of continuous integration).

One way to speed up the execution time of tests is to limit the amount seeding for individual tests. Consider only seeding the absolute minimum that the particular test case requires to run.

You also need to make it easy and straight forward to do the initial seeding. Maintenance of the seed scripts is a large part of maintaining the tests themselves. You need to be able to easily share the guts of the seeding between tests (think seed methods/classes that you reuse in setup) which makes it easier for everyone to seed as well as making them centralized so if something fundamentally changes you only need to update in one place.

I hope this helps. Good luck and test on!