9

My application has big integration with the database. Classes that use the database are very crucial for the system, so I write small unit tests for classes that I call Repository. The reason behind the unit-testing database operations are for example verification of caching or correct order of query execution (to avoid foreign key constraints etc). Plus, I write these tests before the actual code since it allows me to work faster.

For unit testing these operations, I use an in-memory database (H2Database - Java). However, since the schema of the database is quite big, creating the schema in the in-memory database takes about ~4 seconds. For hundreds of unit tests such these, this will make my unit-test suite quite slow.

In order to make it faster, instead of erasing completely the in-memory database between tests, I just truncate all the tables from it and not re-create the schema from scratch. This runs quite faster. Truncation takes about 5-10ms and the database is statically shared across all unit tests. This way, 1 test takes 4 seconds. 100 tests take 5 seconds.

But, (to me) that means that this in-memory database/connection is an important part of the test suite. If this does not work correct, I might end up with wrong feedback from my actual tests. So, in the end, I wrote some unit tests that verify the behavior of the test-database. For example, that all tables are truncated every time you call TestInMemoryDb.create().

The fact I wrote "tests" for the test code, is it a smell? Is it a wrong decision to do? I know that since it gives me confidence, it should be alright. But... is it? Or it is unnecessary thing to do?

There is always the option to create between the tests only the parts of the schema I need. But this comes with one disadvantage. All the CREATE TABLE statements in my unit tests are reducing readability and tests are full of noise. Having the schema as a whole in an sql file and then running it, is more convenient and easier to maintain instead of breaking it into pieces here and there.

11
  • I have never written test for the database, it sounds weird to me. Anyway, it seems to be some code that should be run on every test , most of the unit test frameworks have methods for that instead of creating tests.
    – X.Otano
    Feb 19 at 16:38
  • @X.Otano Sorry but I don't understand your comment. Unit test frameworks have what? Writing tests for classes that use the database is important. Suppose that a class has a fetchUser method. I want to have tests that this class fetched the user of the data and mapped the record to an object correct. user.isPremiumAccount in my business logic is a boolean. But in database is a number 0/1. I want confidence that this mapping was correct etc
    – George Z.
    Feb 19 at 16:44
  • 2
    @GeorgeZ.: See it this way: you did not write tests for your tests, you wrote tests for a tool that is important for your working process, and that tool is complex enough that it deserves tests. (And yes, the tool itself is used for testing purposes, but it may also be used for something different, so don't bother). There is nothing wrong with this approach, quite the opposite. I am sure most testing frameworks have they own unit tests.
    – Doc Brown
    Feb 19 at 22:24
  • 1
    @DocBrown on the other hand that boils down to "my tests are so complex they require I develop a special test library to run them". Sure its good to test a test library you make. but its better to simplify your tests so its not needed
    – Ewan
    Feb 20 at 9:41
  • 1
    @Ewan: it is great when we get all the tools we need directly off-the-shelf. Unfortunately, most complex software systems sooner or later run into situations where they don't have that luxury. When creating testing tools for the own process with care, those can simplify testing (and it seems the OP is in that situation).
    – Doc Brown
    Feb 20 at 13:09

5 Answers 5

13

I disagree that you have written a test for your tests. Consider the reason why you write tests to begin with: to verify non-trivial behavior. What you describe seems like an integration test, even if the code being tested is not used in application code.

Clearly TestInMemoryDb.create() has some complexity. Furthermore, it is a crucial part of the test infrastructure. A failure in this single method could potentially cause hours of debugging. Despite that behavior not being used in the application, these are valuable tests.

As the application evolves, the database schema will change. This will impact the procedure that "cleans" the database and hopefully results in test failures. The assumption is this would save time.

What you did is make a judgement call about one piece of code. It might be a different situation if you decide to write tests for test code as a general matter of practice, regardless of complexity. While, generally, you don't write tests for test code, there is always an exception to the rule. This appears to be a good example of why you should not blindly follow guidelines.

1
  • Exactly. This is what I was thinking. I believe that important code that gets called everywhere (even in tests), should be tested. If one day, InMemoryDbExpectations fails, it will not surprise me if all my DAL-test classes fail as well. I will know where to look for issues. However, I took sometime to post in stackexchange since this is the first time I come against something like this... And all that happens because I dont want my unit tests to take minutes in order to run. I will wait a couple of days before accepting your answer. Thank you for your time.
    – George Z.
    Feb 19 at 20:01
9

I am sure most major testing tools (like all the ones named with "xyz"-Unit, replace "xyz" by your favorite letter from the alphabet) have unit tests. These are tools complex enough to deserve such tests, and they are written by people who value unit testing. For example, I just checked NUnit and JUnit, they are Open Source, hence you can find their unit tests easily in the source tree.

So when you implement your own tools for your test suite, there is nothing special in writing unit tests for your tools, quite the opposite. This only sounds strange because of the wording: the tests you wrote are not for testing "the tests", they are testing your test tool, which is not the same. Hence don't overthink it.

0

I can suggest another way to look at it: by making sure your test DB operates how you expect you're not testing your tests. You're testing your test framework.

Off-the-shelf test frameworks have their own tests. But you've implemented bespoke test functionality according to your specific needs, so you should verify it works how you expect.

-1

I think you are heading down a rabbit hole here.

On the face of it you have written some code that "cleans up" databases, and writing a test for that code makes sense. Build that code and use the binary in your other tests without retesting it like you would any other third party library.

However, I think the common way of testing Data Access Layer code these days is to spin up a verison of the real database in a container and run your tests against that. This illiminates and problems caused by discrepencies between an in memory database and the real version.

When it comes to clearing out tables between tests, I would ask you "do you expect your code to run on non-empty databases?" if so, then the tests should pass without clearing the db down.

Using a real db doesn't fix the problem of "clearing my db takes ages" but it does fix the problem of "I am writing a bunch of code and infrastructure which is only ever used in my tests, may be faulty and requires its own test suite"

17
  • 3
    I don't think testing against a real database fixes the situation. Even if you run tests against a real database, it would still be desirable to "clean" it out. It makes sense to write these tests to verify a real database gets cleaned out as well. Feb 19 at 18:57
  • 1
    Running tests against a database with other test data could cause failures unless test writers take extra care to insert unique data for each test. Feb 19 at 18:59
  • This is why I will write integration tests. To use the real PROD DB and also treat the system as a whole. But I want my tests to be determenistic. E.g when I insert a record using a sequence, I want its ID to be predictable (value 1) in order to make assertions. For unit tests, H2 gives me what I want. Testcontainers is not an option unfortunately :( (licensing issues). For integration tests of course I will use production RDBMS vendor.
    – George Z.
    Feb 19 at 19:55
  • 1
    In the life of your application, inserting data into an empty database happens once. Most of the time your application will work with a database that has been used for a year at least. That needs testing.
    – gnasher729
    Feb 19 at 21:18
  • @gnasher729: it is more efficient to automate functional testing, at that point. I don't think this answer deserves a negative score, but a test run against a database with a bunch of old test data is not likely to mimic production anyhow. Data problems arise because people do crazy things. Carefully curated tests won't catch those conditions. Feb 19 at 23:29
-1

There's "test infrastrcture" and "test code". You don't want to write tests for "test code" in any circumstances, but you do need to write tests for the "test infrastrcture" if it becomes complicated. These tests can perhaps be running a dummy test case, aka smoke test.

Not the answer you're looking for? Browse other questions tagged or ask your own question.