What should I be looking for when evaluating unit testing libraries and tools?

Question

I'm in the process of choosing new unit testing tools for soon to be started project. There are plenty of them!

The basic requirements I have would be that it can run tests on at least Windows and Linux, it is C++/C++03 compatible, it can execute tests in parallel (e.g., to take advantage of multi-core processors), and it is able to be run from the command line. But these are pretty broad requirements, and haven't narrowed down the field.

To try and do that, I'm looking into potential gotchas and pitfalls of different types of implementations. For example, if I wanted to use a unit testing engine in higher level language like Python, would it still produce output formatted like C unit tests? My main intent here is to use the same storage and report tools for both C++ and Python results tests.

I'm also wondering if it's necessary to have a specific engine to run the tests, or is it enough to have them integrated into our continuous integration engine (buildbot).

Are these two concerns above valid? What else should I be concerned about when evaluating unit testing engines?

Answer 1

I have used both Google Test and Boost Test. Both are xUnit frameworks, work on Unix and Windows, and are mature. I ended up using Google Test due to Google Mock since I wanted a mocking framework as well. Either should work well under buildbot.

Personally, I want the unit test framework to be minimal and fast. I want many tests and thus want them to run in a reasonable time frame. I personally like to use the main framework for a language instead of a mega tool but that is personal. If you wanted, you could look at continuous iteration testing: every time you save a file, it auto-runs the test cases that file depends on.

You could always write an Interface class to your unit test thus allowing a quick and effective change of frameworks.

Answer 2

My experiences involve CruiseControl.NET and Team Foundation Build. I'm a .NET developer myself. These come from my experiences, and only include the experiences where we had testing as part of the build process, so if they don't match your environment, sorry.

I'm also wondering if it's necessary to have a specific engine to run the tests, or is it enough to have them integrated into our continuous integration engine

If you are doing a build every check-in, it is possible to have builds triggered before the previous build is finished running. How do you want to handle situations like this? For example, if it takes 30 minutes to run through the entire test suite, and code gets checked in every 15 minutes, do you want to skip a few builds? Skip some tests? Stack them up, build them in order and get back to Bob that he broke the build with his check-in 5 hours ago?

Team Foundation Build can use multiple cores for builds and tests with 2010 (previous versions could only use multiple cores for C++ builds). CruiseControl.NET can run separate threads, but reading the documentation, it appears that each project can be on its own thread, but that you can't have multiple threads per project (I may be wrong). We've never had a multi-core machine for builds in the environments I've worked in, so I can't speak to how good/fair/bad they are (or wrong I am).

At one previous employer, we included Python scripts in the build, but we didn't have any testing set up to test the Python. NANT was used for the .NET components.

I'm also wondering if it's necessary to have a specific engine to run the tests

I never had time to do it, but at a previous employer (we sold "shrinkwrap" software), we had many bugs that were operating system specific (and sometimes service pack specific), so one of my goals was to set up multiple test machines with different OSes (both 32 + 64 bits and all the various flavors of desktop and server Windows from XP onwards). The build machines were simple and older WinXP machines (or virtuals) because that was all the "hardware" we were allowed by the managers. Instead, all tests ran on the build machine. We also set up validation suites as collections of unit tests, although these were not run for minor updates.