24

Currently at my job, we have a large suite of unit tests for our C++ application. However we don't use a unit test framework. They simply utilize a C macro that basically wraps an assert and a cout. Something like:

VERIFY(cond) if (!(cond)) {std::cout << "unit test failed at " << __FILE__ << "," << __LINE__; asserst(false)}

Then we simply create functions for each of our tests like

void CheckBehaviorYWhenXHappens()
{
    // a bunch of code to run the test
    //
    VERIFY(blah != blah2);
    // more VERIFY's as needed
}

Our CI server picks up "unit test failed" and fails the build, emailing the message to the developers.

And if we have duplicate setup code, we simply refactor it like we would any other duplicated code we'd have in production. We wrap it behind helper functions, make some test classes wrap setting up frequently used scenarios.

I know there's frameworks out there like CppUnit and boost unit test. I'm wondering what value do these add? Am I missing what these bring to the table? Is there something useful I could gain from them? I'm hesitant to add a dependency unless it adds real value especially since it seems what we have is dead simple and works well.

1
  • Test frameworks might be good in the sense of providing standardization for integration with other tooling. In reality, the tools probably aren't compatible, and it's easier to write a shell script to integrate something simple like you describe, than find some plugin for whatever system that might provide the standard way of doing it. In general you probably work with slightly higher skill devs who realize the value in just doing something simple. Jul 28, 2021 at 7:01

6 Answers 6

10

As others have already said, you already have your own, simple, home made framework.

It seems to be trivial to make one. However, there are some other features of a unit-test framework that are not so easy to implement, because they take some advanced knowledge of the language. The features I generally require from a test framework and are not so easy to homebrew are:

  • Automatic collection of test cases. I.e. defining new test method should be enough to get it executed. JUnit automatically collects all methods whose names start with test, NUnit has the [Test] annotation, Boost.Test uses the BOOST_AUTO_TEST_CASE and BOOST_FIXTURE_TEST_CASE macros.

    It's mostly convenience, but every bit of convenience you can get improves the chance developers will actually write the tests they should and that they will connect them in correctly. If you have long instructions, somebody will miss part of them now and than and perhaps some tests won't be running and nobody will notice.

  • Ability to run selected test cases, without tweaking code and recompiling. Any decent unit-test framework allows you to specify which tests you want to run on command-line. If you want to debug on unit tests (it's most important point in them for many developers), you need to be able to select just some to run, without tweaking code all over the place.

    Say you just received bug report #4211 and it can be reproduced with unit test. So you write one, but than you need to tell the runner to run just that test, so you can debug what's actually wrong there.

  • Ability to mark tests expected failures, per test case, without modifying the checks themselves. We actually switched frameworks at work to get this one.

    Any decently sized test suite will have tests, that are failing because the features they test were not implemented yet, were not finished yet, nobody had time to fix them yet or something. Without ability to mark tests as expected failures, you won't notice another failure when there are some regularly, so the tests stop serving their main purpose.

5
  • thanks I think this is the best answer. Right now my macro does its job, but I can't do any of the features you mention.
    – Doug T.
    Mar 11, 2012 at 16:17
  • 1
    @Jan Hudec "It's mostly convenience, but every bit of convenience you can get improves the chance developers will actually write the tests they should and that they will connect them in correctly."; All the testing frameworks are (1) nontrivial to install, often have more outdated or nonexhaustive installation instructions than up to date valid instructions; (2) if you commit to a test framework directly, without an interface in the middle, you're married to it, switching frameworks isn't always easy.
    – Dmytro
    Oct 9, 2016 at 18:36
  • @Jan Hudec If we expect to have more people write unit tests, we must have more results on google for "What is a Unit Test", than "What is Unit Testing". There is no point of doing Unit Testing if you have no idea what a Unit Test is independent of any Unit Testing Frameworks or definition of Unit Testing. You cannot do Unit Testing unless you have a strong understanding of what is a Unit Test, as otherwise there is no point of doing Unit Testing.
    – Dmytro
    Oct 9, 2016 at 18:38
  • I don't buy this convenience argument. Writing test code is very hard if you leave the trivial world of examples. All this mockups, setups, libraries, external mockup server programs etc. They all require that you know the test framework from inside out.
    – Lothar
    Dec 30, 2016 at 17:35
  • @Lothar, yes, it is all a lot of work and a lot to learn, but still having to write simple boilerplate over and over again because you lack couple of useful utilities makes the work much less pleasant and that makes noticeable difference in effectivity.
    – Jan Hudec
    Dec 30, 2016 at 22:05
28

Seems like you already use a framework, a home-made one.

What is the added value of more popular frameworks? I would say that the value they add is that when you have to exchange code with people outside your company, you can do it, since it's based on the framework which is known and widely used.

A home-made framework, on the other hand, forces you to either never share your code, or to provide the framework itself, which may become cumbersome with the growth of the framework itself.

If you give your code to a colleague as is, with no explanation and no unit test framework, he would not be able to compile it.

A second drawback of home-made frameworks is the compatibility. Popular unit test frameworks tend to ensure compatibility with different IDEs, version control systems, etc. For the moment, it may not be very important for you, but what will happen if one day you will need to change something in your CI server or migrate to a new IDE or a new VCS? Will you reinvent the wheel?

Last but not least, larger frameworks provide more features you may need to implement in your own framework one day. Assert.AreEqual(expected, actual) is not always enough. What if you need to:

  • measure precision?

    Assert.AreEqual(3.1415926535897932384626433832795, actual, 25)
    
  • void test if it runs for too long? Reimplementing a timeout may not be straightforward even in languages which facilitate asynchronous programming.

  • test a method which expects an exception to be thrown?

  • have a more elegant code?

    Assert.Verify(a == null);
    

    is fine, but isn't it more expressive of your intent to write the next line instead?

    Assert.IsNull(a);
    
5
  • The "framework" we use is all in a very small header file and follows the semantics of assert. So I'm not too worry about the drawbacks you list.
    – Doug T.
    Feb 28, 2012 at 13:56
  • 4
    I consider the asserts the most trivial part of test framework. The runner that collects and runs the test cases and checks results is the non-trivial important part.
    – Jan Hudec
    Feb 28, 2012 at 14:50
  • @Jan I don't quite follow. My runner is a main routine common to every C++ program. Does a unit test framework runner do something more sophisticated and useful?
    – Doug T.
    Feb 28, 2012 at 17:56
  • 1
    Your framework only allows for the semantics of assert and running tests in a main method... so far. Just wait till you have to group your asserts into multiple scenarios, group related scenarios together based on initialized data, etc. Feb 28, 2012 at 21:25
  • @DougT.: Yes, decent unit test framework runner does some more sophisticated useful things. See my full answer.
    – Jan Hudec
    Feb 29, 2012 at 12:53
4

As others have already said, you already have your own, home made framework.

The only reason I can see for using some other test framework would be from an industry "common knowledge" point of view. New developers wouldn't have to learn your home made way (albeit, it looks very simple).

Also, other test frameworks may have more features that you could take advantage of.

1
  • 1
    Agreed. If you're not running into limitations with your current testing strategy, I see little reason to change. A good framework would probably provide better organization and reporting capabilities, but you would have to justify the additional work required to integrate with your code base (including your build system).
    – TMN
    Feb 28, 2012 at 15:00
3

You already have a framework even if it is a simple one.

The main advantages of a bigger framework as I see them is the ability to have many different kind of assertions (such as asserting raises), a logical order to the unit tests, and the ability to run only a sub-set of unit tests at a time. Also, the pattern of xUnit tests are fairly nice to follow if you can -- for example the setUP() and tearDown() one. Of course, that locks you into said framework. Note that some frameworks have better mock integration than others -- google mock and test for example.

How long will it take you to refactor all your unit tests to a new framework? Days or a few weeks maybe worth it but more maybe not so much.

2

The way I see it, you both have the advantage, and you are at a "disadvantage" (sic).

The advantage is that you have a system that you feel comfortable with, and which works for you. You are happy that it confirms the validity of your product, and you would probably find no business value in attempting to change all of your tests for something that uses a different framework. If you can refactor your code, and your tests pick up the changes - or better yet, if you can modify your tests and your existing code fails the tests until it is refactored, then you have all of your bases covered. However...

One of the advantages of having a well designed unit testing API is that there is a lot of native support in most modern IDE's. This won't affect the hard-core VI and emacs users out there who sneer at the Visual Studio users out there, but for those of use who make use of a good IDE, you have the ability to debug your tests and execute them within the IDE itself. This is good, however there is an even greater advantage depending on the framework you use, and that is in the language used to test your code.

When I say language, I'm not talking about a programming language, but instead I am talking about a rich set words wrapped up in a fluent syntax that makes test code read like a story. In particular, I have become an advocate for the use of BDD frameworks. My personal favorite DotNet BDD API is StoryQ, but there are several others with the same basic purpose, which is to take a concept out of a requirements document, and write it in code in a similar way to how it is written in the spec. The really good APIs however go even further, by intercepting every individual statement within a test and indicating whether that statement executed successfully, or failed. This is incredibly useful, as you get to see the entire test executed without returning early, which means your debugging efforts become incredibly efficient as you only need to focus your attention on the parts of the test that failed, without needing to decode the entire call sequence. The other nice thing is that the test output shows you all of this information, both in the console and also with some APIs in a nicely formatted report that you can give to management if they are interested in that sort of thing.

As an example of what I am talking about, compare the following:

Using Asserts:

Assert(variable_A == expected_value_1); // if this fails...
Assert(variable_B == expected_value_2); // ...this will not execute
Assert(variable_C == expected_value_3); // ...and nor will this!

Using a fluent BDD API: (Imagine that the italicized bits are basically method pointers)

WithScenario("Test Scenario")
    .Given(*AConfiguration*) // each method
    .When(*MyMethodToTestIsCalledWith*, variable_A, variable_B, variable_C) // in the
    .Then(*ExpectVariableAEquals*, expected_value_1) // Scenario will
        .And(*ExpectVariableBEquals*, expected_value_2) // indicate if it has
        .And(*ExpectVariableCEquals*, expected_value_3) // passed or failed execution.
    .Execute();

Now granted the BDD syntax is longer, and wordier, and these examples are terribly contrived, however for very complex testing situations where a lot of things are changing in a system as a result of a given system behaviour, the BDD syntax offers you a clear description about what your are testing, and how your test configuration has been defined, and you can show this code to a non-programmer and they will instantly understand what is going on. In addition, if "variable_A" fails the test in both cases, the Asserts example would not execute past the first assert until you had fixed the problem, while the BDD API would execute every method called in the chain, in turn, and indicate which individual parts of the statement were in error.

Personally I find this approach works much better than the more traditional xUnit frameworks do in the sense that the language of testing is the same language as your customers will speak of their logical requirements. Even so, I have managed to use xUnit frameworks in a similar style without needing to invent a complete testing API to support my efforts, and while the asserts will still effectively short-circuit themselves, they read more cleanly. For instance:

Using Nunit:

[Test]
void TestMyMethod()
{
    const int theExpectedValue = someValue;

    GivenASetupToTestMyMethod();

    var theActualValue = WhenIExecuteMyMethodToTest();

    Assert.That(theActualValue, Is.EqualTo(theExpectedValue)); // nice, but it's not BDD
}

If you do decide to explore using a unit testing API, my advice is to experiment with a large number of different APIs for a little while, and to keep and open mind about your approach. While I personally advocate for BDD, your own business needs may require something different for your team's circumstances. The key however is to avoid second-guessing your existing system. You can always support your existing tests with a few tests using another API if needed, but I certainly wouldn't recommend a huge test rewrite just to make everything the same. As legacy code falls out of use, you can easily replace it and its tests with new code, and tests using an alternative API, and this without needing to invest in a major effort that won't necessarily give you any real business value. As for using a unit testing API, if the syntax allows a nice natural language to be used, and if even better, the framework allows your coding/debugging efforts to become more efficient, then you probably can make a good business case for using a suitable unit testing API.

1

What you have is simple and gets the job done. If it works for you, great. You don't need a mainstream unit testing framework, and I would hesitate to go to the work of porting an existing library of unit tests over to a new framework. I think the biggest value of unit testing frameworks is to reduce the barrier to entry; you just start writing tests, because the framework is already in place. You're past that point, so you won't get that benefit.

The other benefit of using a mainstream framework--and it's a minor benefit, IMO--is that new developers may already be up to speed on whatever framework you're using, and so will require less training. In practice, with a straightforward approach like what you've described, this shouldn't be a big deal.

Also, most mainstream frameworks have certain features that your framework may or may not have. These features reduce plumbing code, and make it faster and easier to write test cases:

  • Auto-running of test cases, by using naming conventions, annotations/attributes, etc.
  • Various more specific asserts, so that you don't have to write conditional logic for all of your assertions or catch exceptions to assert their type.
  • Categorization of test cases, so you can easily run subsets of them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.