2

I am not sure if anyone has done it. I am trying to do something that is in general, uncommon i.e. unit-testing executable (Windows) or ELFs (Linux). I know that CppUnit provides a good unit testing facility, but I have never used it for unit-testing (used UnitTest++). I hear rumours that you can unit-test executables too. Does anyone have the experience in this?

A relevant post regarding the philosophy of it was here

  • UnitTest++ is in my experience more convenient to use than CppUnit. IMO the Boost.Test is even better. – Jan Hudec Jun 25 '13 at 8:01
  • @JanHudec Interesting. Nevertheless, you have recommended using scripting language for autmated testing, so I will try that now :) – hagubear Jun 25 '13 at 8:15
  • 1
    Depends on which tests you are automating. For automating unit tests of C++ code you need C++ unit test framework. For automating functional and integration tests I definitely recommend scripting language. – Jan Hudec Jun 25 '13 at 8:48
  • @JanHudec I actually meant the same! I have done unit testing for code. It is the .exe and ELFs that are messy. I need to rehost/retartget them and before I perform actual integration with my own source code, I need to validate those binary executables/ELFs in case they have major mismatch with the agreed product specification and deliverables. The only way to do that is, what you have proposed already, the automated testing by means of scripting language packs like CTest and Expect; if you understand what I am on about! – hagubear Jun 25 '13 at 8:52
  • 2
    Sure. I am just a bit pedantic about the terminology. "automated" means the test cases are programmed and run without need of user interaction, which includes "unit" tests. On the other hand the distinction between "unit", "functional" and "integration" test depends on how big chunks you are testing and each of them can be automated or manual (manual unit tests probably don't make sense, but in languages with interactive interpreter they would certainly be possible). – Jan Hudec Jun 25 '13 at 8:58
5

External processes can be tested (as Doc Brown already mentioned, it will not be unit tests) by test tools in any language. So I recommend using a language where running external tools is easy, so some scripting language rather than C++. Some options are:

  • CTest is part of CMake. It runs external processes with specified parameters and checks their output and return value. The capabilities are rather limited, but it is useful to eventually combine tests written with various tools. And integrate them in the build if you use CMake.
  • Unix Shell (using cygwin or msys on Windows). You can use ShUnit or write a bit of custom glue, in shell it is simple.
  • Expect is a tool specially designed for testing command-line programmes. It supports emulating terminal to the tested application, so it can be used for testing applications that require terminal interaction, which can't be tested with ctest or shell.
  • There is a Perl version of Expect and a wrapper to make simple uses simpler, Expect::Simple, or you can use Perl with just it's standard Test::Harness.
  • Python also has pexpect module for the same purpose and has the advantage of being better integrated in Windows.
  • I am sure Ruby will have such tool as well.
  • There are even some tools that can automate interaction with GUI applications, like the Linux Desktop Testing Project (which since learned to control applications using native Windows and MacOS GUI toolkits too).

You will likely use respective "xUnit" framework with the above mentioned languages and you could equally well do it with C++. It will handle running the tests in sequence for you, but the testing tools it provides won't be of much use for you. You'll have to write your own assertions for checking the output or use of the expect modules.


Note, that some of the above tools exist much longer than any xUnit. First version of expect was created in 1987 and DejaGNU (testing framework on top of expect used by GCC and some other GNU projects) changelog goes back to 1992 while I could only trace SUnit (Smalltalk, apparently first unit-testing framework) back to 1997.

Good example of project with extensive functional tests over command-line interface is gcc. It has a huge collection of code snippets that should compile and a huge collection of code snippets that should not compile and the test suite tries to compile them with the compiler that was just built and verifies whether the results work where they should and that appropriate diagnostics is given for those that shouldn't compile. Most other compilers and interpreters have similar test suites.

Another example would be git, where the primary interface is also command-line and tests are done using custom shell framework utilizing the runner from perl Test module, while it does not have much toward C-level API that would allow running tests in unit-test style.

  • +1, that is a good addition to my answer (though running external tools using C++ is IMHO not really hard). – Doc Brown Jun 25 '13 at 7:38
  • @DocBrown: No, it's not difficult, but getting their output and searching it for regular expressions gets a bit tedious. And scripting languages save some time when you tune the test input and patterns by not requiring compilation. – Jan Hudec Jun 25 '13 at 7:50
  • I agree 100%. In comparison, I got my experiences from using NUnit for running "exe" tests. It had the advantage that we did not have to add another test driving tool to our environment (we were using NUnit already for unit testing), or introduce a new programming language (things I would suggest the OP also to consider). RegExp support was also fine, and compile times for a small test DLL is neglectable. But as I wrote, testing "exe" files will be possible with almost any unit testing tool, and surely with any of the tools you suggested. – Doc Brown Jun 25 '13 at 11:38
2

You can use almost any xUnit test tool for automatic testing of executables (however, this will not result in unit tests, but integration or system tests). This assumes your executable can be run in some batch mode without user interaction, and it will produce some machine-readable output for which you can check if it is the expected output or not.

Just write a test which runs your executable in a separate process (see, for example, here), pipes the output into a file and add a function to your test lib which compares the content of the file with a second "expected output" file. You may have to add additional safety measures like time-out testing if you expect your executables to crash or leave batch-mode in case of a bug.

Running the separate process is easy, testing the output, however, can be arbitrary complex, dependent on your program-under-test and the complexity if its output. But the latter part is nothing you can expect any xUnit tool to help you with.

  • Interesting. Thanks for pointing that out. I am in the interservice/industry training and sims. area and have got things like rehosting/retargeting of .exe and elf files on my rehost machine. It is a pain to test them before integration. I guess a good way to go about it is vaccum integration i.e. treat it as an integration test but actually perform unit tests on it. (I think I sound stupid now!) – hagubear Jun 25 '13 at 7:17
  • @hagubear: You can't perform unit test on something else than units (which here means smallest separable pieces of code down to individual functions and classes). It will be simply automated test, that will be the level of functional and integration test. – Jan Hudec Jun 25 '13 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.