Acceptance testing with preloaded data or filling it on the go

Question

We're figuring out whether we should write acceptance tests that revolves around preloaded data on a database file, or programmatically added needed data per test, or filling database while testing other application features.

This is a more detailed, easy use case: you want to acceptance test (or UI-test) adding a Book to the system. But in order to do that, you need to have Category rows in the database already, so you can assign one to the book. As you're writing your test case for adding a book, you:

1. Use a previously seeded database file that has already categories in it.

   • cons would be you need to keep that file up with every change in the code, with the latests changes on the system, AND you'd need to fill it while considering additional business logic (i.e. maybe you seeded authors with random parameters; but turns out an author can't be selectable unless it passes x, y and z conditions; and the seeder didn't take that in consideration)

2. Programmatically add the category in the test.

   • cons would be you need to keep up with additional business logic, again.

3. Each tested feature retains data, so you first run acceptance test for adding categories, and then for adding a book.

   • cons would be your testing tool would need not to isolate tests, or at least run them in a certain order.

Maybe there's a 4th option I haven't considered yet? What's the correct approach for this common task?

Answer 1

We are doing it second way. The best will be to show it by example (python):

def test_that_we_can_add_book_to_library(self):
    category = Category('Fantasy')
    self.library.add_category(Category('Fantasy'))
    book = Book(Title('Hobbit'), category)
    self.library.add_book(book)
    self.library.has_book(book)

What is important here is a fact that library is an object created for tests only. It has behaviours that encapsulates logic (that can change) needed for tests. Additionaly it makes test more readable. library.add_category is such behaviour, if anything changes in business logic related to category, then we change it only there.

library.has_book is an assert method that checks if book was added to our storage.

It is really well described in the book "Growing object-oriented software: Guided by tests". Everyone that wants to write good tests should read it imho.

Answer 2

I prefer doing both.

Keep a small predefined database or set of fixtures that can be used for smoke tests. It should have the bare minimum of data -- a single user, a single category, a single book, etc. This lets you write and run at least some basic tests without much extra work.

Then, for more complex tests they should create the data they need on the fly (and clean it up when done!).

Another solution is to start with a bare database, and write tests that populate it with a core set of objects. So the workflow is:

• stand up new database
• run the setup tests which populate the database with core data
• if the setup succeeds, run additional tests which can use this initial setup data, plus create any additional data it needs on the fly.

The advantage to this second approach is that you have a single process that can act both as a smoke test of parts of your UI or API, and act as a setup step that prepares the database for more extensive testing.