In my job we work in a data driven web application and performance is a important business feature. So, queries in the model tier are frequently large and contain much of the business logic. As a result we have many Integrations Tests that rely in XML datasets used by our test framework(an abstraction to DbUnit) to populate data model. We have migration scripts containing DDL and data domain changes that are executed before all tests, but the framework clean all tables before the execution of each test. We are considering changing this behavior and do not clean domain tables anymore. We would continue defining application data specific to the tests in each XML dataset, but regarding the domain data:
Should we continue defining it in each XML dataset too?
Pros:
- Tests will not break in case of domain data changes that do not impact business logic.
- We can define a small subset of domain data sufficient to test business logic.
Cons:
- It may be necessary create too many tables and rows to represent all relationships that the test need.
- It would be expensive to fix datasets in case of changes in the data model.
- Data duplication in many XML datasets in the project.
Should we do not clean the domain data inserted by the scripts? These scripts would execute only one time before all Integration Tests and the data would be common to all tests.
Pros:
- Tests will break in case of domain data changes that impact business logic.
- It would be easier to create data to the test, because we only need to create application data.
Cons:
- Tests may break in case of domain data changes that do not impact business logic. We should be careful to not test things like number of rows returned or text attributes of domain entities.
- If to test some business logic we need specific characteristics of the domain data that we cannot assume it is true in production, we need to change the actual data domain before the execution of the test and rollback in the end. But how change something that is mutable and unpredictable?
