When writing tests for one piece of software, say a library, do you prefer to compile all unit tests into one, or separate them into several executables?

The reason I'm asking is because I am currently using CUnit to test a library I'm working on. The tests are split up into separate suites that are compiled into one executable complete with printed output for failures. Now, the build system for that library is CMake (which, despite its name, has little to do with CUnit), which comes with its own testing framework, CTest. CTest allows me to register a list of executables that serve as tests.

I'm pondering whether to use CTest for automated testing runs. However, this would require me to split up the tests I've written so far into separate compile targets. Otherwise, I can't really utilize some of CTests advanced features, such as selectively running tests.

I realize this is more a question of what tools to use and their handling and conventions, but apart from that, are there any other reasons to prefer a single test executable over separate ones? Or vice versa?

  • Separate them into separate executables by class. Each class should have it's own unit test unless the class is not unit testable and thus needs to be tested by other classes indirectly. – Brian Jun 14 '12 at 18:06
  • 1
    If you have a large library with hundreds of classes, each with a unit test, the build time is much longer if you create a complete binary for each, compared to one (or a few) big binaries. Also, that's a lot of makefiles to manage, each with individual link lines. – JBRWilkinson Jun 14 '12 at 23:44
  • It's not that big. Less than 20 modules. I'm also able to compile them all with the same flags, so CMake can generate the Makefiles without much work on my part. – Benjamin Kloster Jun 15 '12 at 6:27

I like to have my automated tests in individual binaries, or at least clumped per "belongs-together" group, and then call them from a simple shell script (where a non-zero exit code signals failure, and output on stderr may be captured to record an explanation). This way, I retain full flexibility on the testing - I can run individual tests directly from the command line, I can make all sorts of fancy scripts if I want to, I can reorder them as I see fit without recompiling anything, etc.

But more importantly, it also allows me to include tests written in different languages or using different toolchains in the same run. For example, the unit tests I write are most likely in the project's main language, and running them is a matter of building and invoking the binaries; but I also want to test my database, and I might want to feed SQL scripts directly to the database for this; I might want to run some static code analysis tool on my code (even if it's just some sort of linter). I may want to run my static HTML through a validity checker. I could run a grep command over the codebase to check for suspicious constructs, coding style violations, or "red-flag" keywords. The possibilities are endless - if it can be run from the command line and adheres to "zero exit status means OK", I can use it.

| improve this answer | |
  • The language agnostic argument is a very good point, since I plan on implementing python bindings for the library down the road. Thanks! – Benjamin Kloster Jun 14 '12 at 17:50
  • @tdammers: any particular testing framework? – JBRWilkinson Jun 14 '12 at 23:47
  • @JBRWilkinson: Just a 30-line shell script. For most of the languages I use, I do have little libraries around for common testing tasks such as running a function, comparing the result to an expected value, and throwing when they don't match. But any given unit testing framework could easily be integrated into such a "meta system", as long as it can run from the command line and signals success/failure through its exit status. – tdammers Jun 15 '12 at 6:20

I tend to have one library for the unit tests of one application (or for a package of libraries that is commonly shared). Within that library, I try to replicate or approximate the namespaces of the objects under test for the test fixtures (I use NUnit mostly). This simplifies compilation, as in .NET there is an overhead inherent in building each binary that would increase the build time of a 20-project solution over that of a 10-project solution with the same LOC. Test binaries aren't distributed anyway, so any organization of the tests into binaries is for your own convenience, and I generally find that YAGNI applies here as anywhere.

Now, I usually don't have the considerations that tdammers has; my code is virtually all in one language, and any test involving SQL strings is not a unit test (unless you're testing that a query-producer returns the expected SQL string given certain criteria), and I virtually never unit-test the actual UI (in many situations it's simply impossible). I also use a unit-testing library that is well-accepted by third-party tools such as build-bots and IDE plugins, and so any concerns of running individual tests, partial suites etc are minimal.

| improve this answer | |
  • 1
    A matter of culture, I guess - in a .NET environment, I would probably reason much like you do. – tdammers Jun 14 '12 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.