The way of running automated tests (specifically, a large number of small, «atomic» unit and regression checks) that I am used to is to maintain a monolithic executable built on top of a test framework that aggregates the individual checks into a test tree and provides «quality of life» features like colourful or machine-readable reporting and selection of tests by name, regular expression or sub-tree.
I was recently acquainted with another way: to have every single check's report as a separate build target in a build system (such as
dune) and make it depend on relevant files of the source code of the project, so that a report must be regenerated exactly when the relevant sources change. The promise is that this would provide a seemingly magical ability for the test suite to execute in the shortest amount of time required to cover a given modification, given that dependencies between the source files and the check reports are wired correctly. The down sides that I anticipate and fear are that:
- It is not trivial to provide «quality of life» features such as those mentioned above. It seems that the only way to get this is to make every check adhere to some standard report form that is understood by the build system acting as a test runner, so a standard must be developed and implemented for all the varieties of checks present in a project — possibly written in different languages and adhering to different principles. Check names and hierarchy should also be communicated to the build system in some way — possibly requiring patching or extending said build system.
- Some relevant checks may not run due to dependencies not being wired correctly. Of course we would still run all checks before merging a branch to assure quality, but giving relevant responses to a programmer in real time is also a design goal, so it is essential to be able to run only those checks that may possibly fail given the diff.
- A test run for a feature close to the root of the dependency tree would require to re-run almost all checks. This would reduce response time dramatically. So, if a very large number of checks would be affected by an edit, it is desirable to be able to tell the build system to intelligently select the most relevant or swift checks — for example, unit tests and not integration, or unit tests pertaining to the edited source file and not those pertaining to source files dependent on the edited source file.
Does this practice have a name? Is it used at large scale in well-known open source projects? Are my fears unfounded? Where can I read more?