1

Is it a good idea to have individual/unique data sets for each integration test or should all tests reuse the same data? My idea for having individual data sets for each test is to have more control for each test to make it easier to update old tests as well as implementing new data. Instead of making sure the new data for a new test is not interfering with the old data/tests I can just add a new data set that will only be used by my new test. To me it makes sense but when reading up on integration/service testing it seems like most (all?) is using the same data for all tests.

My biggest reason for having unique data sets for each tests is because I want to write tests for a microservice architecture and making sure you get unique ID's that is the same throughout all DB's turned out to be a bit messy. If I would have individual datasets I could follow AAA:

// Assign
Mock up database, could be CSV files loaded into an in memory-db.

// Act
Make a call to the endpoint I want to test, could be done through for example MSTest, Webtest or POSTMAN

// Assert
Make sure that the response contains the data I wanted.

In the case above, the CSV would then be the data sets. So each test would get individual CSV-files that would be used to seed the DB prior to running the test.

7
  • I don't understand. How would test-specific data give you more control and easier updating? Feb 8, 2018 at 9:30
  • For the same reason I use AAA for unit testing. This was a question, not a statement. Please feel free to answer why you don't think it is a good idea.
    – Oskar
    Feb 8, 2018 at 9:48
  • The more edge cases you test, the better. So if you are going to the trouble of creating a new data set, it would make sense to apply that data set to all tests (where appropriate). I'd go for "have multiple data sets and use all of them for all tests".
    – David Arno
    Feb 8, 2018 at 11:17
  • That makes sense @DavidArno. So let's say I have a test for retrieving all active users then when I want to write a new test for retrieving all inactive users then I put the data for that test in a new dataset but when I run my tests I load all datasets into my test-database? My only concern would be that I think it would be a real PITA when you have hundreds of tests to try to come up with new unique ID's. That is actually my main reason for having individual datasets in the first place, it get's a bit messy in a microservice architecture, lots of DB's with ID's that must match.
    – Oskar
    Feb 8, 2018 at 12:09
  • My biggest reason for having unique data sets for each tests is because I want to write tests for a microservice architecture and making sure you get unique ID's that is the same throughout all DB's turned out to be a bit messy. -- Why isn't that sufficient justification? Feb 8, 2018 at 16:26

2 Answers 2

2

It depends completely of the situation:

If you use unique test data sets:

  • you are less subject to be impacted by previous test runs, if the tested system has persistance.
  • you are also less subject to interference if several tests are run in parallel
  • it's easier to trace data when results are not satisfactory
  • you need more effort to generate each test set, because you have to ensure consistency of all the data
  • you test in a bubble, and you might miss stupid mistakes, such as some basic concurrency issues (for example when data remains locked but shouldn't)

If you use same data:

  • the reference data might be altered in an unfavorable manner by a previous test.
  • it's more difficult when a test fails to find out if it's due to the current test run or some previous issues
  • forget about testing concurrency consistency
  • it's easier to prepare test sets.
1

I would opt for individual data per test.

Technically, it's not important. If you always start from a predefined state on each test, it does not matter to the computer.

However, in the unlikely (cough) event that a test fails, unique data makes it so much easier for the developers to communicate. It's way easier to remember what the test with "Karl & Karen" was about, then the test #54 with Alice. "Karl & Karen" was the nice hypothetical couple with the Golden Retriever where Karl never reads his emails on time and Karen reminds him about it and now both of their accounts have the same email address and it should work anyway... Alice #54? No idea what that was about. It failed? Wow. I need to read the documentation.

2
  • I updated my question now, I think we misunderstood each other. My idea was to have for example different CSV-files (or JSON or XML, whatever floats your boat) to use for seeding the DB's for each test. So instead of having one huge CSV-file per table that all tests use I have multiple smaller CSV-files where each test has it's own file. So for example in the folder SameEmail I have CSV-files for seeding the test with Karl & Karen and in other files I have Ken & Darren that has the same phone number etc. I hope it makes sense!
    – Oskar
    Feb 8, 2018 at 14:58
  • 1
    @Oskar Sure. I'm not sure where the misunderstanding is. Computers don't care. Humans do. My answer remains the same.
    – nvoigt
    Feb 8, 2018 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.