How to setup data for service tests

Question

We are trying to work out the best way to setup test data when we run our isolated functional tests. These tests test just the service and mock out other dependencies and use an in memory database. Each test will require some data to be setup and there are two thoughts about how this setup is done:

Inject the data directly into the database. The complaint here is that this is brittle and ties the tests to the structure of the database.

The alternative proposed is to use the services API to insert data as part of the test setup. The issue with this is that the scope of test has expanded to include the logic of inserting the data as well as the actual test case itself.

Personally I favour injecting into the database but as there is no clear consensus here I'd like to see what others think!

Answer 1

In short, all other things being equal, injecting the data into the database is the better approach. It reduces tight coupling between the separate business logic features. It also

I've struggled with the same question not too long ago. For reference, I finally convinved the PM to let me implement a basic amount of unit testing for a project that has been developed with no testability in mind.

The data in our project is shaped by a series of operations - the project is an application which performs these many operations. However, there is so much generated (meta)data in after the first X steps, that it became nigh impossible for me to generate the actual data state I needed to test (and in some cases there were 100+ combinations of data sets to test).
This was further compounded by the fact that the data structure was continually changing because of a bad and vague analysis and an equally vague project owner.

There are a lot of flaws in the current project management. The analysis should not have been vague. The code needed to be developed with testability in mind. The entire data structure should have been revisited instead of copy/pasted from the last similar project.

Nonetheless, this is the project I was writing unit tests for, so I had to face the facts. There were ulterior reasons (not directly related to my development) which made the proper approach (data injection) nigh impossible to get right.

At this point, I decided to therefore rely on the business logic to arrange the state of the database.
Am I happy about it? No. It feels so very dirty. However, I do have to admit that it does actually work given the less-than-perfect situation the project already finds itself in.

The biggest drawback here is that a small error in one of the first operations will cause many tests to fail, since almost all subsequent operation logic depends on that first operation to be correct.
But due to the linear nature of the operations, it's still relatively easy to spot which test is the cause. We actually ended up numbering our tests so that the order of operations (and thus the order of tests depending on each other) became obvious.

By default, I do not recommend choosing to rely on the business logic to set your data state. However, there are some situations where the data is so complicated and you're unable to actually change that fact. In those situations, it may be beneficial to simply have the logic generate the data state for you simply to ensure that you can have some tests instead of none at all.

Answer 2

It all depends on the kind of tests you are intending to perform.

• For unit tests of the data-access/data-source layer, data injected directly in the database is the best approach. With precise control on the test data, you reduce the potential cause of errors to the tested object itself. In addition, you can verify that your data-source behaves as expected with incomplete or inconsistent data.

• For any other tests, i.e. unit tests of the business logic, or integration tests:

  • db injection could still be a valid approach, but if and only if the data model is simple.
  • on the contrary, db injection could be counter productive with a complex data model, because you introduce the risk of unexpected inconsistencies in the test data that could be falsely interpreted as error in the tested component.

So for complex data models (and data-source tests excepted), the better approach would be to create test data using the API. This works very well if using the following mitigation measures:

• ensure that the data creation API is designed in a way to facilitate test automation (for instance, the onion architecture sees the testing components as integral part of the architecture)
• make sure that the unit tests of the writing API is unit tested first, so that the writing errors could be isolated and would not pollute the dependent tests
• if the data is very long to be produced, the you could imagine a mixed approach: create the heavy test data once. Create a snapshot of the database. Restart tests from the snapshot (i.e. it's as if you would inject data, but without the risk of accidental inconsistencies). continue the remaining tests using API if necessary.

This gives you the best of both worlds (even if you'd be developing an ERP systems with thousands of interrelated tables and tricky binary data types).