Can a build system be used effectively as a unit test runner?

Question

The way of running automated tests (specifically, a large number of small, «atomic» unit and regression checks) that I am used to is to maintain a monolithic executable built on top of a test framework that aggregates the individual checks into a test tree and provides «quality of life» features like colourful or machine-readable reporting and selection of tests by name, regular expression or sub-tree.

I was recently acquainted with another way: to have every single check's report as a separate build target in a build system (such as make or dune) and make it depend on relevant files of the source code of the project, so that a report must be regenerated exactly when the relevant sources change. The promise is that this would provide a seemingly magical ability for the test suite to execute in the shortest amount of time required to cover a given modification, given that dependencies between the source files and the check reports are wired correctly. The down sides that I anticipate and fear are that:

• It is not trivial to provide «quality of life» features such as those mentioned above. It seems that the only way to get this is to make every check adhere to some standard report form that is understood by the build system acting as a test runner, so a standard must be developed and implemented for all the varieties of checks present in a project — possibly written in different languages and adhering to different principles. Check names and hierarchy should also be communicated to the build system in some way — possibly requiring patching or extending said build system.
• Some relevant checks may not run due to dependencies not being wired correctly. Of course we would still run all checks before merging a branch to assure quality, but giving relevant responses to a programmer in real time is also a design goal, so it is essential to be able to run only those checks that may possibly fail given the diff.
• A test run for a feature close to the root of the dependency tree would require to re-run almost all checks. This would reduce response time dramatically. So, if a very large number of checks would be affected by an edit, it is desirable to be able to tell the build system to intelligently select the most relevant or swift checks — for example, unit tests and not integration, or unit tests pertaining to the edited source file and not those pertaining to source files dependent on the edited source file.

Does this practice have a name? Is it used at large scale in well-known open source projects? Are my fears unfounded? Where can I read more?

Answer 1

Sbt has testQuick that only runs tests that failed before, weren't run before, or that a dependency was recompiled. It doesn't require any manual specifying of rules. I imagine other build systems have (or could get) something similar.

However, it's mostly useful for rapid TDD-style development, where you're running your tests several times per minute, and are acutely aware of what you just changed. You still run all your tests before you commit and push, because coupling can happen in weird ways that don't involve directly-shared code.

Answer 2

• It is not trivial to provide «quality of life» features such as those mentioned above

  It is absolutely trivial.

  Each individual compiled test is built with, or linked against, some unit testing framework that provides the desired features.

  You don't write a new top-level test runner for each unit, you just re-use your generic test runner with a different subset of tests.

  In your edit you added

  Check names and hierarchy should also be communicated to the build system in some way — possibly requiring patching or extending said build system.

  But I can't think of any reason why you'd want that.

  The build system must already know the names of the build targets for each affected test, because you rely on it to build them.

  It doesn't also need to know the symbolic name of each test - that will just appear in the (parseable, or human-readable, or whatever) output.

• Some relevant checks may not run due to dependencies not being wired correctly.

  Normally you only run individual tests when doing an incremental build, locally. You're still going to run all unit tests when your CI system does a complete build.

  Anyway, if your build system has missing dependencies, how can you trust it to correctly re-compile your non-test code? Either you trust your build artefacts (including tests), or your build system is broken and you have much bigger problems.

• A test run for a feature close to the root of the dependency tree would require to re-run almost all checks

  How is that different? You changed something that affects many tests, so you re-run those tests. Is re-building your monolithic test runner somehow faster than re-building all the affected tests it contains?

  ... it is desirable to be able to tell the build system to intelligently select the most relevant or swift checks — for example, unit tests and not integration.

  Yes, but we're only discussing unit tests. Integration tests are out of scope for the question, and I wouldn't expect them to be triggered automatically in a local build.

  By default no unit-test system (driven by the build system) should be running integration tests. Because they're not unit tests.