1

The way of running automated tests (specifically, a large number of small, «atomic» unit and regression checks) that I am used to is to maintain a monolithic executable built on top of a test framework that aggregates the individual checks into a test tree and provides «quality of life» features like colourful or machine-readable reporting and selection of tests by name, regular expression or sub-tree.

I was recently acquainted with another way: to have every single check's report as a separate build target in a build system (such as make or dune) and make it depend on relevant files of the source code of the project, so that a report must be regenerated exactly when the relevant sources change. The promise is that this would provide a seemingly magical ability for the test suite to execute in the shortest amount of time required to cover a given modification, given that dependencies between the source files and the check reports are wired correctly. The down sides that I anticipate and fear are that:

  • It is not trivial to provide «quality of life» features such as those mentioned above. It seems that the only way to get this is to make every check adhere to some standard report form that is understood by the build system acting as a test runner, so a standard must be developed and implemented for all the varieties of checks present in a project — possibly written in different languages and adhering to different principles. Check names and hierarchy should also be communicated to the build system in some way — possibly requiring patching or extending said build system.
  • Some relevant checks may not run due to dependencies not being wired correctly. Of course we would still run all checks before merging a branch to assure quality, but giving relevant responses to a programmer in real time is also a design goal, so it is essential to be able to run only those checks that may possibly fail given the diff.
  • A test run for a feature close to the root of the dependency tree would require to re-run almost all checks. This would reduce response time dramatically. So, if a very large number of checks would be affected by an edit, it is desirable to be able to tell the build system to intelligently select the most relevant or swift checks — for example, unit tests and not integration, or unit tests pertaining to the edited source file and not those pertaining to source files dependent on the edited source file.

Does this practice have a name? Is it used at large scale in well-known open source projects? Are my fears unfounded? Where can I read more?

12
  • The way you have phrased the question makes it opinionated. What one person sees as best practice another person will see as bad practice. There is an interesting question lurking in here though about good ways to selectively run tests based on source code changes, and another good question about maintenance of the linkage from test to source. A third good question is whether a partial test run is sufficient to verify a given set of source. But which one are you asking?
    – Kain0_0
    Nov 10, 2020 at 22:44
  • @Kain0_0 A brief look around convinced me that «opinionated» questions are acceptable. See for example softwareengineering.stackexchange.com/q/341208 and softwareengineering.stackexchange.com/q/237342 — they are well up voted and I do not see how my question is different. I must be wrong? Nov 10, 2020 at 22:50
  • Test automation such as you describe has been commonly available for a long time in Build/CI tools such as Jenkins, GitHub CI, TeamCity, etc. In most cases there's no need to reinvent the wheel by creating your own test runner. Nov 10, 2020 at 23:00
  • @BenCottrell But I think those features are rather crude? That is to say, they only track dependencies between large components, on the level of whole packages and test suites. What I have in mind is a system that tracks individual source code files within a single project. Nov 10, 2020 at 23:08
  • @Kain0_0  I removed some verbiage that I guess may be objectionable. Looks better? Nov 10, 2020 at 23:22

2 Answers 2

3

Sbt has testQuick that only runs tests that failed before, weren't run before, or that a dependency was recompiled. It doesn't require any manual specifying of rules. I imagine other build systems have (or could get) something similar.

However, it's mostly useful for rapid TDD-style development, where you're running your tests several times per minute, and are acutely aware of what you just changed. You still run all your tests before you commit and push, because coupling can happen in weird ways that don't involve directly-shared code.

2
  • You are saying: «Sbt has testQuick that only runs tests that … a dependency was recompiled». What is understood as a dependency here — a single source file or a package as a whole? How fine-grained can it be? Nov 10, 2020 at 23:37
  • I believe a class, but I'm not sure. Nov 10, 2020 at 23:46
3
  • It is not trivial to provide «quality of life» features such as those mentioned above

    It is absolutely trivial.

    Each individual compiled test is built with, or linked against, some unit testing framework that provides the desired features.

    You don't write a new top-level test runner for each unit, you just re-use your generic test runner with a different subset of tests.

    In your edit you added

    Check names and hierarchy should also be communicated to the build system in some way — possibly requiring patching or extending said build system.

    But I can't think of any reason why you'd want that.

    The build system must already know the names of the build targets for each affected test, because you rely on it to build them.

    It doesn't also need to know the symbolic name of each test - that will just appear in the (parseable, or human-readable, or whatever) output.

  • Some relevant checks may not run due to dependencies not being wired correctly.

    Normally you only run individual tests when doing an incremental build, locally. You're still going to run all unit tests when your CI system does a complete build.

    Anyway, if your build system has missing dependencies, how can you trust it to correctly re-compile your non-test code? Either you trust your build artefacts (including tests), or your build system is broken and you have much bigger problems.

  • A test run for a feature close to the root of the dependency tree would require to re-run almost all checks

    How is that different? You changed something that affects many tests, so you re-run those tests. Is re-building your monolithic test runner somehow faster than re-building all the affected tests it contains?

    ... it is desirable to be able to tell the build system to intelligently select the most relevant or swift checks — for example, unit tests and not integration.

    Yes, but we're only discussing unit tests. Integration tests are out of scope for the question, and I wouldn't expect them to be triggered automatically in a local build.

    By default no unit-test system (driven by the build system) should be running integration tests. Because they're not unit tests.

7
  • I suppose there is some misunderstanding. I expanded on the points — I hope it is more clear now why they are important and not trivial to address. Nov 11, 2020 at 20:32
  • Ah, I think I see the confusion. It is usual to have incremental build-driven unit tests run locally when the developer runs an incremental build, to provide instant feedback. This output can simply be displayed directly, there's no real use for formatting a big report document for the whole project.
    – Useless
    Nov 12, 2020 at 14:16
  • When you run a full build in CI, if you want a pretty report, it would be up to the CI script to take all the output from the build & unit tests (or the test report artefacts) and form it into a single document. So, the hierarchy proceeds either from the directory hierarchy or is somehow expressed in the content.
    – Useless
    Nov 12, 2020 at 14:18
  • I've honestly never wanted any report that wasn't either "all tests succeeded" or the list of tests that failed and their output, so I may have missed the language in your comment about the importance of displaying results in a hierarchy.
    – Useless
    Nov 12, 2020 at 14:20
  • I certainly want a pretty report in all cases, and you seem to agree that it is not a feature usually found in a build system. At the very least, the build system must be aware that some tasks are checks and treat them differently — I am unsure if, for example, make is configurable enough to be up to the task. I do not see how a CI script enters the picture — certainly it is not in a better position to format a report than the build system that actually runs the checks. Nov 12, 2020 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.