How should C++ Unit Test code be organized for maximum Unit Test efficiency?

Question

• This question is not about Unit Testing Frameworks.
• This question is not about writing Unit Tests.
• This question is about where to put the UT code written and how/when/where to compile and run it.

In Working Effectively with Legacy Code, Michael Feathers asserts that

good unit tests ... run fast

and that

A unit test that takes 1/10th of a second to run is a slow unit test.

I think these definitions do make sense. _{I also think that they imply that you have to keep a set of Unit Tests and a set of Those Code Tests That Take Longer separately, but I guess that's the price you pay for only calling something a Unit Test if it runs (very) fast.}

Obviously the problem in C++ is that to "run" your Unit Test(s), you have to:

1. Edit your code (production or Unit Test, depending on which "cycle" you're in)
2. Compile
3. Link
4. Start Unit Test Executable(s)

---

Edit _{(after weird close vote)}: Before going into the details, I'll try summarize the point here:

How can C++ Unit Test code be effectively organized, so that it's both efficient to edit the (test) code and to run the test code?

---

The first problem then is to decide where to put the Unit Test code so that:

• it's "natural" to edit and view it in combination with the associated production code.
• it's easy/quick to start the compilation cycle for the unit your currently changing

The second, related, problem then is what to compile so that the feedback is instantaneous.

Extreme options:

• Each Unit-Test-Test-Unit lives in a separate cpp file and this cpp file is compiled+linked separately (together with the source code unit file it tests) to a single executable which then runs this one Unit Test.
  • (+) This minimizes startup (compile+link!) time for the single Test Unit.
  • (+) The test runs super fast, because it only tests one unit.
  • (-) Executing the whole suite will need to start a bazillion of processes. Can be a problem to manage.
  • (-) The overhead of process starts will become visible
• The other side would be to have -- still -- one cpp file per test, but all the test cpp files (together with the code they test!) are linked into one executable (per module / per project / pick your choice).
  • (+) Compile time still would be OK, as only changed code will compile.
  • (+) Executing the whole suite is easy, as there's just one exe to run.
  • (-) The suite will take ages to link, as each recompilation of any object will trigger a re-link.
  • (-)(?) The suit will take longer to run, although if all Unit Tests are fast, the time should be OK.

So, how are real world C++ Unit Tests handled? If I only run that stuff nightly/hourly, the second part doesn't really matter, but the first part, namely how to "couple" the UT code to the production code, so that it's "natural" for developers to keep both in focus always matters I think. (And if developers have the UT code in focus, they'll want to run it, which brings us back to part two.)

Real world stories and experience appreciated!

_Notes:

• _{This question intentionally leaves unspecified platform and make/project system.}
• _{Questions Tagged UT & C++ is a great place to start, but unfortunately too many questions, and especially answers, are too heavily focused on the details or on specific frameworks.}
• _{A while ago, I answered a similar question on structure for boost unit tests. I find this structure to be lacking for "real", fast Unit Tests. And I find the other question too narrow, hence this new question.}

Answer 1

We have all unit tests (for a module) in one executable. The tests are put into groups. I can execute a single test (or some tests) or a group of tests by specifying a (test/group) name on the command line of the test runner. The build system can run group "Build", the test department can run "All". The developer can put some tests into a group like "BUG1234" with 1234 being the issue tracker number of the case he's working on.

Answer 2

First, I disagree with "1) Edit your (production) code and your Unit Test". You should modify only one at a time, otherwise if the result changes you won't know which one caused it.

I like to put unit tests in a directory tree that shadows the main tree. If I have /sources/componentA/alpha/foo.cc and /objects/componentA/beta/foo.o, then I want something like /UTest_sources/componentA/alpha/test_foo.cc and /UTest_objects/componentA/beta/test_foo.o. I use the same shadow tree for stub/mock objects and whatever other sources the tests need. There will be some edge cases, but this scheme simplifies things a lot. A good editor macro can pull up the test source alongside the subject source effortlessly. A good build system (e.g. GNUMake) can compile both and run the test with one command (e.g. make test_foo), and can manage a bazillion such processes -- just the ones whose sources have changed since they were last tested -- quite easily (I have never found the overhead of starting these processes to be a problem, it's O(N)).

In the same framework, you can have larger-scale tests (no longer unit tests) which link many objects together and run many tests. The trick is to sort these tests by how long they take to build/run, and work them into your daily schedule accordingly. Run the one-second-or-less test whenever you feel like it; start the ten-second test and stretch; five-minute test and take a break; half-hour test and go for lunch; six-hour test and go home. If you find that you're wasting a lot of time e.g. relinking a huge test after changing only one small file, you're doing it wrong-- even if the linking were instantaneous, you'd still be running a long test when it wasn't called for.