Your question is very difficult to provide a general answer for. Also, your choice of tech-stack greatly affects how to approach this. E.g., Ruby on Rails is extremely opinionated, so following these practices in that framework could easily lead to something very unidiomatic. Also, the use of an ORM often has some effect on the interface to the data access layer (which is the primary reason I don't use an ORM).
But you mention separate parts of the application:
- Controllers
- Service layer for business logic
- Database layer (I think you implied that conversion went into this
layer, an approach I support)
This distinction is a very good starting point in my experience. This also lends itself well to the "Clean Architecture".
The different layers have different responsibilities, and how you would test it differs from layer to layer. But generally, your test should focus on the responsibility of that layer.
Business logic
If you follow the "Clean Architecture", this would be divided into functions or classes (depending on your programming paradigm), each representing a concrete use case. Each use case would receive parameters for that use case. E.g., if the use case is "transfer money", the parameters could be the "debit account no", "credit account no", "amount", etc.
The return value of a use case would then be some object representing the outcome. This could contain information about if this was successful or not, and relevant information that the caller would want back. E.g., if the use case creates a new entity in the system, the caller might want the ID, or some kind of reference to the new entity.
If you start by writing tests for this layer, you have a good chance that your tests describes actual business needs. I prefer to test this layer as a whole, mocking out the database layer. By doing so, you are letting your business rules define the needs of a database layer (i.e., define the interface).
Note: Most developers I've worked with (which is quite a few) don't like testing this layer as a whole (they argue that it isn't unit testing), and separate testing of the database save/load from testing the domain logic. In my experience that approach however, leads to a lot more tests that doesn't provide much value, and more often needs to be changed when code is refactored. I have also often seen this leads to excessive mocking, and in some cases the developers had accidentally mocked out the very thing that should have been tested.
Database layer
Now that your tests for the business logic defined what the interface for your database layer needs to be, you can start writing this functionality test first. When testing this layer, I would always use a real database.
My tests for this layer would normally be some kind of round trip tests, something that inserts data in the database, loads it again, and see that we get the same thing out as we put in. For queries, I would insert data that should be found by the query, and data that should be ignored by the query, and then verify that the correct results are returned.
Always write these tests in a way that they don't depend on existing data. Some years ago, it was quite popular to have standard fixture data for tests, but I would strongly advise against that. Let the tests control explicitly everything that affects the outcome of the test.
Controller layer
Again, if following the clean architecture, the responsibility of this layer is basically to receive an HTTP request, convert the URL parameters and request body into a call to a use case. The outcome of the use case should then be translated into an HTTP response. This is basically true no matter if it's an API or server generated pages.
So to test this layer, I would mock the use case implementations, and verify that the HTTP request is converted to the correct use case request, and that the use case response is converted to the correct HTTP response.
Notice that I express my tests using HTTP requests and responses. That is how I would test this layer. Most often, I see tests for this layer that calls functions on controllers. But if you test this on controller functions, your tests are coupled to an implementation detail.
The system I am building right now uses Express.js to serve the API, but I could change that to something else without modifying a single test. In fact, my tests would tell me if such a change was implemented correctly.
You ability to express the tests in terms of HTTP requests and responses depends on your tech stack. With Node.js and go, it's quite easy. In .NET and .NET Core, it's very difficult, though Nancy seems to support this style of testing.
In Node.js I use the supertest library and in Go, you have something build in.
Don't forget to refactor
TDD is not just about writing tests first, it the "red, green, refactor" cycle.
In my experience, TDD works because it provides
- Fast feedback
- A safety net for refactoring.
And by constantly refactoring, you are encouraging yourself to not over-engineer in the first place.
So a good measure of you test suite is, how often do you need to change your tests when you refactor your code.
So this is my general approach to applying TDD when writing backend code. Many will disagree to this approach. Some say my tests are not unit tests, but integration tests. But I don't find that distinction useful, what matters is that I get fast feedback, and that I can refactor safely.
