CppUnit for unit-testing executable files?

Question

I am not sure if anyone has done it. I am trying to do something that is in general, uncommon i.e. unit-testing executable (Windows) or ELFs (Linux). I know that CppUnit provides a good unit testing facility, but I have never used it for unit-testing (used UnitTest++). I hear rumours that you can unit-test executables too. Does anyone have the experience in this?

A relevant post regarding the philosophy of it was here

Answer 1

External processes can be tested (as Doc Brown already mentioned, it will not be unit tests) by test tools in any language. So I recommend using a language where running external tools is easy, so some scripting language rather than C++. Some options are:

• CTest is part of CMake. It runs external processes with specified parameters and checks their output and return value. The capabilities are rather limited, but it is useful to eventually combine tests written with various tools. And integrate them in the build if you use CMake.
• Unix Shell (using cygwin or msys on Windows). You can use ShUnit or write a bit of custom glue, in shell it is simple.
• Expect is a tool specially designed for testing command-line programmes. It supports emulating terminal to the tested application, so it can be used for testing applications that require terminal interaction, which can't be tested with ctest or shell.
• There is a Perl version of Expect and a wrapper to make simple uses simpler, Expect::Simple, or you can use Perl with just it's standard Test::Harness.
• Python also has pexpect module for the same purpose and has the advantage of being better integrated in Windows.
• I am sure Ruby will have such tool as well.
• There are even some tools that can automate interaction with GUI applications, like the Linux Desktop Testing Project (which since learned to control applications using native Windows and MacOS GUI toolkits too).

You will likely use respective "xUnit" framework with the above mentioned languages and you could equally well do it with C++. It will handle running the tests in sequence for you, but the testing tools it provides won't be of much use for you. You'll have to write your own assertions for checking the output or use of the expect modules.

---

Note, that some of the above tools exist much longer than any xUnit. First version of expect was created in 1987 and DejaGNU (testing framework on top of expect used by GCC and some other GNU projects) changelog goes back to 1992 while I could only trace SUnit (Smalltalk, apparently first unit-testing framework) back to 1997.

Good example of project with extensive functional tests over command-line interface is gcc. It has a huge collection of code snippets that should compile and a huge collection of code snippets that should not compile and the test suite tries to compile them with the compiler that was just built and verifies whether the results work where they should and that appropriate diagnostics is given for those that shouldn't compile. Most other compilers and interpreters have similar test suites.

Another example would be git, where the primary interface is also command-line and tests are done using custom shell framework utilizing the runner from perl Test module, while it does not have much toward C-level API that would allow running tests in unit-test style.

Answer 2

You can use almost any xUnit test tool for automatic testing of executables (however, this will not result in unit tests, but integration or system tests). This assumes your executable can be run in some batch mode without user interaction, and it will produce some machine-readable output for which you can check if it is the expected output or not.

Just write a test which runs your executable in a separate process (see, for example, here), pipes the output into a file and add a function to your test lib which compares the content of the file with a second "expected output" file. You may have to add additional safety measures like time-out testing if you expect your executables to crash or leave batch-mode in case of a bug.

Running the separate process is easy, testing the output, however, can be arbitrary complex, dependent on your program-under-test and the complexity if its output. But the latter part is nothing you can expect any xUnit tool to help you with.