Should you write unit tests for templates and frameworks, such as in .NET?

Question

How much (if any) unit tests should be written when you start a new templated project? In particular, let's take as an example a typical .net framework template for asp.net mvc or other similare ones that give a bunch of boilerplate.

Typically, much of this is pre-configured configuration, and you delete much of the "implementation" it templates. But should that configuration be unit tested? Should you make sure that App.UseMVC is called, that Authentication is wired up, or that various objects are wired up in dependency injection?

A lot of people will say "What does it hurt?" and maybe they're right, but then I think about maintenance. When I add a new configuration to the dependency try it's relatively easy to query for it, but then i'm testing implementation details rather than design. It's also not so easy to catch if you've forgotten to add a configuration, since most unit tests are not using "live" dependency configurations (nor should they).

I struggle with what to test in frameworks in general. Especially scaffolded frameworks. What about testing if logging is being called, or that specific attibutes are applied?

What is good practice in this regard?

Answer 1

A lot of people will say "What does it hurt?" and maybe they're right, but then I think about maintenance.

I agree. Testing is good. Rigorous testing is better. But testing can be an infinite well of effort, and you get diminished returns as your test cases get more niche and fringe.

There is a reasonable line to draw, but not everyone draws it the same way, nor should it be drawn the way regardless of context (budget, scope, scale, ...)

I struggle with what to test in frameworks in general.

Generally speaking, you don't test frameworks (or libraries, for that matter). The assumption is that your external dependencies work as advertised; the onus of testing is put on the developers of said external dependency.

Testing focuses on your business logic. This is not to be confused with the business layer, where a lot (but not all) of your business logic lives. In this context, business logic refers to specific behavior that you implemented in your application.

Should you make sure that App.UseMVC is called, that Authentication is wired up, or that various objects are wired up in dependency injection?

I have never come across tests whose specific focus are any of these things you listed. However, endpoint integration tests would generally catch these kinds of issues as they would render the endpoints unusable.

For example, if your application requires authentication; then you've defined some behaviors on how to handle (un)authenticated users, both in the authentication endpoints themselves and the other endpoints (for which you need to be authenticated to access). Therefore, by testing the behavior of those endpoints, you should inherently be testing that the application behaves correctly when the user is/isn't authenticated.

I suspect that the things you're mostly thinking of right now are (indirect) integration test materials.

What about testing if logging is being called, or that specific attibutes are applied?

You need to distinguish here between the application behavior set forth by your requirements and any technical implementation. In most cases, tests tend to focus on the behavior specified in the requirements. In other words: does the application behave the way the customer needs it to?

If by logging you mean the kind of behind-the-scenes logging that isn't part of the application's visible behavior, but rather something that is implemented for the developers to be able to trace issues when they occur, then this is generally not tested.

However, in some cases logging may be a functional requirement (e.g. for auditing purposes). In such a case, the logging is considered a behavior, and therefore it warrants testing.

---

To summarize, in general you will find that:

• Unit tests focus on behavior of components you wrote; not of the framework.
• Integration tests focus on the behavior that is defined in your requirements. By definition of what integration testing is; this also tests and verifies the glue that holds your components together. Whether that glue is related to a framework, boilerplated, or handwritten by you, is irrelevant as to the integration tests themselves.
• You don't test your external frameworks and libraries. Exceptions can be made in cases where the added cost of doing so far outweighs the cost of issues slipping by. E.g. I'm sure NASA tests all of their suppliers' code to make triple sure everything is up to snuff; but this is generally speaking well out of scope for your everyday business application.

Answer 2

You don't need to test that framework - except in two cases:

1. If you reasonably suspect that it doesn't do what it is supposed to do.

2. If you suspect that what it is supposed to do is not what you think it is supposed to do.

Especially in the second case, a unit test to make sure that what it does is what you think it should do can be valuable.