Do I really need a unit test framework?

Question

Currently at my job, we have a large suite of unit tests for our C++ application. However we don't use a unit test framework. They simply utilize a C macro that basically wraps an assert and a cout. Something like:

VERIFY(cond) if (!(cond)) {std::cout << "unit test failed at " << __FILE__ << "," << __LINE__; asserst(false)}

Then we simply create functions for each of our tests like

void CheckBehaviorYWhenXHappens()
{
    // a bunch of code to run the test
    //
    VERIFY(blah != blah2);
    // more VERIFY's as needed
}

Our CI server picks up "unit test failed" and fails the build, emailing the message to the developers.

And if we have duplicate setup code, we simply refactor it like we would any other duplicated code we'd have in production. We wrap it behind helper functions, make some test classes wrap setting up frequently used scenarios.

I know there's frameworks out there like CppUnit and boost unit test. I'm wondering what value do these add? Am I missing what these bring to the table? Is there something useful I could gain from them? I'm hesitant to add a dependency unless it adds real value especially since it seems what we have is dead simple and works well.

Answer 1

As others have already said, you already have your own, simple, home made framework.

It seems to be trivial to make one. However, there are some other features of a unit-test framework that are not so easy to implement, because they take some advanced knowledge of the language. The features I generally require from a test framework and are not so easy to homebrew are:

• Automatic collection of test cases. I.e. defining new test method should be enough to get it executed. JUnit automatically collects all methods whose names start with test, NUnit has the [Test] annotation, Boost.Test uses the BOOST_AUTO_TEST_CASE and BOOST_FIXTURE_TEST_CASE macros.

  It's mostly convenience, but every bit of convenience you can get improves the chance developers will actually write the tests they should and that they will connect them in correctly. If you have long instructions, somebody will miss part of them now and than and perhaps some tests won't be running and nobody will notice.

• Ability to run selected test cases, without tweaking code and recompiling. Any decent unit-test framework allows you to specify which tests you want to run on command-line. If you want to debug on unit tests (it's most important point in them for many developers), you need to be able to select just some to run, without tweaking code all over the place.

  Say you just received bug report #4211 and it can be reproduced with unit test. So you write one, but than you need to tell the runner to run just that test, so you can debug what's actually wrong there.

• Ability to mark tests expected failures, per test case, without modifying the checks themselves. We actually switched frameworks at work to get this one.

  Any decently sized test suite will have tests, that are failing because the features they test were not implemented yet, were not finished yet, nobody had time to fix them yet or something. Without ability to mark tests as expected failures, you won't notice another failure when there are some regularly, so the tests stop serving their main purpose.

Answer 2

Seems like you already use a framework, a home-made one.

What is the added value of more popular frameworks? I would say that the value they add is that when you have to exchange code with people outside your company, you can do it, since it's based on the framework which is known and widely used.

A home-made framework, on the other hand, forces you to either never share your code, or to provide the framework itself, which may become cumbersome with the growth of the framework itself.

If you give your code to a colleague as is, with no explanation and no unit test framework, he would not be able to compile it.

A second drawback of home-made frameworks is the compatibility. Popular unit test frameworks tend to ensure compatibility with different IDEs, version control systems, etc. For the moment, it may not be very important for you, but what will happen if one day you will need to change something in your CI server or migrate to a new IDE or a new VCS? Will you reinvent the wheel?

Last but not least, larger frameworks provide more features you may need to implement in your own framework one day. Assert.AreEqual(expected, actual) is not always enough. What if you need to:

• measure precision?

  Assert.AreEqual(3.1415926535897932384626433832795, actual, 25)
  

• void test if it runs for too long? Reimplementing a timeout may not be straightforward even in languages which facilitate asynchronous programming.

• test a method which expects an exception to be thrown?

• have a more elegant code?

  Assert.Verify(a == null);
  

  is fine, but isn't it more expressive of your intent to write the next line instead?

  Assert.IsNull(a);
  

Answer 3

As others have already said, you already have your own, home made framework.

The only reason I can see for using some other test framework would be from an industry "common knowledge" point of view. New developers wouldn't have to learn your home made way (albeit, it looks very simple).

Also, other test frameworks may have more features that you could take advantage of.

Answer 4

You already have a framework even if it is a simple one.

The main advantages of a bigger framework as I see them is the ability to have many different kind of assertions (such as asserting raises), a logical order to the unit tests, and the ability to run only a sub-set of unit tests at a time. Also, the pattern of xUnit tests are fairly nice to follow if you can -- for example the setUP() and tearDown() one. Of course, that locks you into said framework. Note that some frameworks have better mock integration than others -- google mock and test for example.

How long will it take you to refactor all your unit tests to a new framework? Days or a few weeks maybe worth it but more maybe not so much.

Answer 5

The way I see it, you both have the advantage, and you are at a "disadvantage" (sic).

The advantage is that you have a system that you feel comfortable with, and which works for you. You are happy that it confirms the validity of your product, and you would probably find no business value in attempting to change all of your tests for something that uses a different framework. If you can refactor your code, and your tests pick up the changes - or better yet, if you can modify your tests and your existing code fails the tests until it is refactored, then you have all of your bases covered. However...

One of the advantages of having a well designed unit testing API is that there is a lot of native support in most modern IDE's. This won't affect the hard-core VI and emacs users out there who sneer at the Visual Studio users out there, but for those of use who make use of a good IDE, you have the ability to debug your tests and execute them within the IDE itself. This is good, however there is an even greater advantage depending on the framework you use, and that is in the language used to test your code.

When I say language, I'm not talking about a programming language, but instead I am talking about a rich set words wrapped up in a fluent syntax that makes test code read like a story. In particular, I have become an advocate for the use of BDD frameworks. My personal favorite DotNet BDD API is StoryQ, but there are several others with the same basic purpose, which is to take a concept out of a requirements document, and write it in code in a similar way to how it is written in the spec. The really good APIs however go even further, by intercepting every individual statement within a test and indicating whether that statement executed successfully, or failed. This is incredibly useful, as you get to see the entire test executed without returning early, which means your debugging efforts become incredibly efficient as you only need to focus your attention on the parts of the test that failed, without needing to decode the entire call sequence. The other nice thing is that the test output shows you all of this information, both in the console and also with some APIs in a nicely formatted report that you can give to management if they are interested in that sort of thing.

As an example of what I am talking about, compare the following:

Using Asserts:

Assert(variable_A == expected_value_1); // if this fails...
Assert(variable_B == expected_value_2); // ...this will not execute
Assert(variable_C == expected_value_3); // ...and nor will this!

Using a fluent BDD API: (Imagine that the italicized bits are basically method pointers)

WithScenario("Test Scenario")
    .Given(*AConfiguration*) // each method
    .When(*MyMethodToTestIsCalledWith*, variable_A, variable_B, variable_C) // in the
    .Then(*ExpectVariableAEquals*, expected_value_1) // Scenario will
        .And(*ExpectVariableBEquals*, expected_value_2) // indicate if it has
        .And(*ExpectVariableCEquals*, expected_value_3) // passed or failed execution.
    .Execute();

Now granted the BDD syntax is longer, and wordier, and these examples are terribly contrived, however for very complex testing situations where a lot of things are changing in a system as a result of a given system behaviour, the BDD syntax offers you a clear description about what your are testing, and how your test configuration has been defined, and you can show this code to a non-programmer and they will instantly understand what is going on. In addition, if "variable_A" fails the test in both cases, the Asserts example would not execute past the first assert until you had fixed the problem, while the BDD API would execute every method called in the chain, in turn, and indicate which individual parts of the statement were in error.

Personally I find this approach works much better than the more traditional xUnit frameworks do in the sense that the language of testing is the same language as your customers will speak of their logical requirements. Even so, I have managed to use xUnit frameworks in a similar style without needing to invent a complete testing API to support my efforts, and while the asserts will still effectively short-circuit themselves, they read more cleanly. For instance:

Using Nunit:

[Test]
void TestMyMethod()
{
    const int theExpectedValue = someValue;

    GivenASetupToTestMyMethod();

    var theActualValue = WhenIExecuteMyMethodToTest();

    Assert.That(theActualValue, Is.EqualTo(theExpectedValue)); // nice, but it's not BDD
}

If you do decide to explore using a unit testing API, my advice is to experiment with a large number of different APIs for a little while, and to keep and open mind about your approach. While I personally advocate for BDD, your own business needs may require something different for your team's circumstances. The key however is to avoid second-guessing your existing system. You can always support your existing tests with a few tests using another API if needed, but I certainly wouldn't recommend a huge test rewrite just to make everything the same. As legacy code falls out of use, you can easily replace it and its tests with new code, and tests using an alternative API, and this without needing to invest in a major effort that won't necessarily give you any real business value. As for using a unit testing API, if the syntax allows a nice natural language to be used, and if even better, the framework allows your coding/debugging efforts to become more efficient, then you probably can make a good business case for using a suitable unit testing API.

Answer 6

What you have is simple and gets the job done. If it works for you, great. You don't need a mainstream unit testing framework, and I would hesitate to go to the work of porting an existing library of unit tests over to a new framework. I think the biggest value of unit testing frameworks is to reduce the barrier to entry; you just start writing tests, because the framework is already in place. You're past that point, so you won't get that benefit.

The other benefit of using a mainstream framework--and it's a minor benefit, IMO--is that new developers may already be up to speed on whatever framework you're using, and so will require less training. In practice, with a straightforward approach like what you've described, this shouldn't be a big deal.

Also, most mainstream frameworks have certain features that your framework may or may not have. These features reduce plumbing code, and make it faster and easier to write test cases:

• Auto-running of test cases, by using naming conventions, annotations/attributes, etc.
• Various more specific asserts, so that you don't have to write conditional logic for all of your assertions or catch exceptions to assert their type.
• Categorization of test cases, so you can easily run subsets of them.