5

We have a system with about 250 tables. The first-time-setup include running about 50 queries (known also as seeds).

We're trying out strategies to automatize our acceptance tests, and our first choice was to clean and rebuild the database on every scenario (aka: every test), so that we could keep them isolated from the rest: meaning a "list all resources" scenario wouldn't check for a resource created in a previously ran "create" scenario.

I'm not sure, whether this "isolation" will bring any measurable benefit, but it does make each scenario execution time longer (about 20 seconds each scenario).

Is it considered a bad practice, or will it bring any known scalability issue, if we change our strategy to do this first-time-setup only at the beginning of the setup phase of the tests execution (or maybe at the beginning of each test suite) rather than on every scenario?

If it helps, we're testing using Behat (the official PHP version of Cucumber).

| improve this question | |
  • 3
    Have you considered doing a backup of the DB following the first setup, and then restore/replace it prior to each test? – RomanK Nov 13 '16 at 23:04
  • @RomanK I had assumed it would take the same time as if I set it up via ORM console command. Maybe I was wrong? – Christopher Francisco Nov 14 '16 at 14:02
  • It is a faster operation usually, though obviously without good knowledge of your environment it's hard to say. It's at least worth trying and measuring. – RomanK Nov 14 '16 at 15:40
  • To echo @RomanK, backup and restore can often be a simple file copy rather than table creation and load. – Alan Shutko Dec 14 '16 at 4:21
3

This is not by any means a bad practice. It may seem like an overkill but doing integration/system type of testing such a way gives you tremendous benefits.

Tests become a lot less fragile when you set them up this way. Imagine you add a new test only to find out that all the other tests that run afterward now fail. This tend to happen because those other tests depended on a previous state that you now changed. What do you do now?

  1. You can either change the way you write your new test so it does not change the state for the consequent tests
  2. You update all other tests behind the new one

Both options are pretty terrible. Option #1 constrains you too much so instead of writing a test that actually verifies the use case fully, you write a test where you only kind of testing a feature. Option #2 will take forever and leaves you with a very sour taste in your mouth.

If you apply the strategy where tests do not depend on previous state, you only have to run that single test knowing with a very high probability that it will run fine as part of the entire suite. Therefore it takes a lot less time from development to full integration. The ability to run a single test on its own makes it a lot easier to isolate and debug issues. The context which a developer needs to hold in their brain is also minimized this way making it easier to focus on the task at hand. To summarize it will take a lot less time and also a lot more fun to develop and maintain such tests.

As you mentioned one negative to setting up the state before each test is that the overall running of the suite takes a longer time. I agree this is annoying and you have to consider how often these tests need to run. In my teams acceptance type of tests only ran at most once a day but not every check in and they had their own dedicated build (in other words, they were NOT part of continuous integration).

One way to speed up the execution time of tests is to limit the amount seeding for individual tests. Consider only seeding the absolute minimum that the particular test case requires to run.

You also need to make it easy and straight forward to do the initial seeding. Maintenance of the seed scripts is a large part of maintaining the tests themselves. You need to be able to easily share the guts of the seeding between tests (think seed methods/classes that you reuse in setup) which makes it easier for everyone to seed as well as making them centralized so if something fundamentally changes you only need to update in one place.

I hope this helps. Good luck and test on!

| improve this answer | |
  • In other words, your TDD/BDD practice doesn't involves acceptance tests, but instead, they have a different purpose? – Christopher Francisco Nov 14 '16 at 17:56
  • Why would it not be an acceptance test? – c_maker Nov 14 '16 at 18:29
  • You mentioned you run acceptance test once a day, while TDD/BDD involves the red/green/refactor cycle which is done many times a minute – Christopher Francisco Nov 14 '16 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.