13

When writing tests for one piece of software, say a library, do you prefer to compile all unit tests into one, or separate them into several executables?

The reason I'm asking is because I am currently using CUnit to test a library I'm working on. The tests are split up into separate suites that are compiled into one executable complete with printed output for failures. Now, the build system for that library is CMake (which, despite its name, has little to do with CUnit), which comes with its own testing framework, CTest. CTest allows me to register a list of executables that serve as tests.

I'm pondering whether to use CTest for automated testing runs. However, this would require me to split up the tests I've written so far into separate compile targets. Otherwise, I can't really utilize some of CTests advanced features, such as selectively running tests.

I realize this is more a question of what tools to use and their handling and conventions, but apart from that, are there any other reasons to prefer a single test executable over separate ones? Or vice versa?

Improve this question
4
  • Separate them into separate executables by class. Each class should have it's own unit test unless the class is not unit testable and thus needs to be tested by other classes indirectly.
    – Engineer2021
    Jun 14, 2012 at 18:06
  • 1
    If you have a large library with hundreds of classes, each with a unit test, the build time is much longer if you create a complete binary for each, compared to one (or a few) big binaries. Also, that's a lot of makefiles to manage, each with individual link lines.
    – JBRWilkinson
    Jun 14, 2012 at 23:44
  • It's not that big. Less than 20 modules. I'm also able to compile them all with the same flags, so CMake can generate the Makefiles without much work on my part.
    – Benjamin Kloster
    Jun 15, 2012 at 6:27
  • Even 20 modules can increase the link time immensely. I went from one per subsystem to one total. I would never even consider one per class.
    – Lothar
    Jan 22, 2022 at 3:16

3 Answers 3

Reset to default
6

I like to have my automated tests in individual binaries, or at least clumped per "belongs-together" group, and then call them from a simple shell script (where a non-zero exit code signals failure, and output on stderr may be captured to record an explanation). This way, I retain full flexibility on the testing - I can run individual tests directly from the command line, I can make all sorts of fancy scripts if I want to, I can reorder them as I see fit without recompiling anything, etc.

But more importantly, it also allows me to include tests written in different languages or using different toolchains in the same run. For example, the unit tests I write are most likely in the project's main language, and running them is a matter of building and invoking the binaries; but I also want to test my database, and I might want to feed SQL scripts directly to the database for this; I might want to run some static code analysis tool on my code (even if it's just some sort of linter). I may want to run my static HTML through a validity checker. I could run a grep command over the codebase to check for suspicious constructs, coding style violations, or "red-flag" keywords. The possibilities are endless - if it can be run from the command line and adheres to "zero exit status means OK", I can use it.

Improve this answer
4
  • The language agnostic argument is a very good point, since I plan on implementing python bindings for the library down the road. Thanks!
    – Benjamin Kloster
    Jun 14, 2012 at 17:50
  • @tdammers: any particular testing framework?
    – JBRWilkinson
    Jun 14, 2012 at 23:47
  • @JBRWilkinson: Just a 30-line shell script. For most of the languages I use, I do have little libraries around for common testing tasks such as running a function, comparing the result to an expected value, and throwing when they don't match. But any given unit testing framework could easily be integrated into such a "meta system", as long as it can run from the command line and signals success/failure through its exit status.
    – tdammers
    Jun 15, 2012 at 6:20
  • I can't see how this matters to the one executable class if you just can add one or two start parameters on the command line. I wrote my test with control flow in the opposite direction. The setup/teardown functions are calling the scripts not the other way around.
    – Lothar
    Jan 22, 2022 at 3:20
3

I tend to have one library for the unit tests of one application (or for a package of libraries that is commonly shared). Within that library, I try to replicate or approximate the namespaces of the objects under test for the test fixtures (I use NUnit mostly). This simplifies compilation, as in .NET there is an overhead inherent in building each binary that would increase the build time of a 20-project solution over that of a 10-project solution with the same LOC. Test binaries aren't distributed anyway, so any organization of the tests into binaries is for your own convenience, and I generally find that YAGNI applies here as anywhere.

Now, I usually don't have the considerations that tdammers has; my code is virtually all in one language, and any test involving SQL strings is not a unit test (unless you're testing that a query-producer returns the expected SQL string given certain criteria), and I virtually never unit-test the actual UI (in many situations it's simply impossible). I also use a unit-testing library that is well-accepted by third-party tools such as build-bots and IDE plugins, and so any concerns of running individual tests, partial suites etc are minimal.

Improve this answer
1
  • 1
    A matter of culture, I guess - in a .NET environment, I would probably reason much like you do.
    – tdammers
    Jun 14, 2012 at 18:58
0

I moved from a relative low 20 something number of executables (one per subsystem) to a single one where i autogenerate a dispatcher that turns command line parameters into a way to call the individual scripts.

I did it because it's the only real way to test NDK Android apps at the moment. The IDE is just restricted. And also it's much better on other IDE's like XCode and QtCreator that i use because the code refactoring is better and there is no danger that some tests are left behind uncompilable for a longer time. Yes uncommenting testcases meet a deadline is common.

In the end i found no real big benefits between doing the one or the other once the build/test process is automated except what your IDE/build system supports best. And adding it all into one is the least common denominator.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.