21

TL;DR

Writing good, useful tests is hard, and has a high cost in C++. Can you experienced developers share your rationale on what and when to test?

Long story

I used to do test-driven development, my whole team in fact, but it didn't work well for us. We have many tests, but they never seem to cover the cases where we have actual bugs and regressions - which usually occur when units are interacting, not from their isolated behaviour.

This is often so hard to test on the unit level that we stopped doing TDD (except for components where it really speeds up development), and instead invested more time increasing the integration test coverage. While the small unit tests never caught any real bugs and were basically just maintenance overhead, the integration tests have really been worth the effort.

Now I've inherited a new project, and am wondering how to go about testing it. It's a native C++/OpenGL application, so integration tests are not really an option. But unit testing in C++ is a bit harder than in Java (you have to explicitely make stuff virtual), and the program isn't heavily object oriented, so I can't mock/stub some stuff away.

I don't want to rip apart and OO-ize the whole thing just to write some tests for the sake of writing tests. So I'm asking you: What is it I should write tests for? e.g.:

  • Functions/Classes that I expect to change frequently?
  • Functions/Classes that are more difficult to test manually?
  • Functions/Classes that are easy to test already?

I began to investigate some respectful C++ code bases to see how they go about testing. Right now I'm looking into the Chromium source code, but I'm finding it hard to extract their testing rationale from the code. If anyone has a good example or post on how popular C++ users (guys from the committee, book authors, Google, Facebook, Microsoft, ...) approach this, that'd be extra helpful.

Update

I have searched my way around this site and the web since writing this. Found some good stuff:

Sadly, all of these are rather Java/C# centric. Writing lots of tests in Java/C# is not a big problem, so the benefit usually outweights the costs.

But as I wrote above, it's more difficult in C++. Especially if your code base is not-so-OO, you have to severely mess things up to get a good unit test coverage. For instance: The application I inherited has a Graphics name space that is a thin layer above OpenGL. In order to test any of the entities - which all use its functions directly - I'd have to turn this into an interface and a class and inject it in all the entities. That's just one example.

So when answering this question, please keep in mind that I have to make a rather big investment for writing tests.

Improve this question
5
  • 3
    +1 for the difficulty in unit testing C++. If your unit test requires you to change the code, don't.
    – DPD
    Commented Jul 19, 2012 at 9:02
  • 2
    @DPD: I'm not so sure, what if something is really worth testing? In the current code base, I can hardly test anything in the simulation code because it all calls the graphics functions directly, and I cannot mock/stub them. All I can test right now are utility functions. But I agree, changing the code to make it "testable" feels... wrong. TDD proponents often say that this will make all your code better in all imaginable ways, but I humbly disagree. Not everything needs an interface and several implementations.
    – futlib
    Commented Jul 19, 2012 at 9:15
  • Let me give you a recent example: I spent one whole day trying to test one singe function (written in C++/CLI) and the test tool MS Test would always crash for this test. It seemed to have some problem with plain CPP references. Instead I just tested the ouptput of its calling function and it worked fine. I wasted a whole day to UT one function. That was a loss of precious time. Also I could not get any stubbing tool suitable to my needs. I did manual stubbing wherever possible.
    – DPD
    Commented Jul 19, 2012 at 9:24
  • That's just the kind of stuff I'd like to avoid :D I guess we C++ devs have to be especially pragmatic about testing. You did end up testing it, so I guess that's OK.
    – futlib
    Commented Jul 19, 2012 at 9:41
  • @DPD: I've thought some more about this, and I think you're right, the question is what kind of tradeoff I want to make. Is it worth refactoring the whole graphic system to test a couple of entities? There wasn't any bug in there I know of, so probably: No. If it starts to feel buggy, I'll write tests. Too bad I can't accept your answer because it's a comment :)
    – futlib
    Commented Jul 20, 2012 at 11:14

3 Answers 3

Reset to default
5

Well, Unit Testing is only one part. Integration tests help you with the problem of your team. Integration Tests can be written for all kinds of applications, also for native and OpenGL applications. You should check out "Growing Object Oriented Software Guided by Tests" by Steve Freemann and Nat Pryce (e.g. http://www.amazon.com/Growing-Object-Oriented-Software-Guided-Signature/dp/0321503627). It leads you step by step through the development of an application with GUI and network communication.

Testing Software that was not test driven is another story. Check Michael Feathers "Working Effectively with Legacy Code" (http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052).

Improve this answer
5
  • I know both books. The things are: 1. We don't want to go with TDD, because it didn't work well with us. We do want tests, but not religiously. 2. I'm sure that integration tests for OpenGL applications are possible somehow, but it takes too much effort. I want to keep improving the app, not start a research project.
    – futlib
    Commented Jul 19, 2012 at 6:58
  • 1
    Be aware that unit testing "after the fact" is always harder then "test first", because the code does not get designed to be testable (reuseable, maintainable etc.). If you want to do it anyway, try to stick to Michael Feathers tricks (e.g. seams) even if you avoid TDD. E.g. if you want to extend a function try something like "sprout method" and try to keep the new method testable. It is possible to do, but harder IMHO.
    – EricSchaefer
    Commented Jul 19, 2012 at 9:53
  • I agree that non TDD code is not designed to be testable, but I wouldn't say it isn't maintainable or reusable per se - as I commented above, some things just don't need interfaces and multiple implementations. Not a problem at all with Mockito, but in C++, I need to make all functions I want to stub/mock virtual. Anyway, non-testable code is my biggest problem right now: I have to change some very fundamental things to make some parts testable, and therefore I want a good rationale on what to test, to make sure it's worth it.
    – futlib
    Commented Jul 19, 2012 at 10:07
  • You're right of course, I'll be careful to make any new code I write testable. But it won't be easy, with the way things work in this code base right now.
    – futlib
    Commented Jul 19, 2012 at 10:08
  • When you add a feature/function, just think about how you could test it. Could you inject any ugly dependencies? How would you know that the function does what it is supposed to do. Could you observe any behavior? Are there any result you could check for correctness? Are there any invariants you can check?
    – EricSchaefer
    Commented Jul 19, 2012 at 10:23
2

It's a shame TDD "didn't work well for you." I think that's the key to understanding where to turn. Revisit and understand how TDD didn't work, what could you have done better, why was there difficulty.

So, of course your unit tests didn't catch the bugs you found. That's kind of the point. :-) You didn't find those bugs because you prevented them from happening in the first place by giving thought to how the interfaces should work and how to make sure they were tested properly.

To answer, you question, as you have concluded, unit testing code that is not design to be tested is difficult. For existing code it may be more effective to use a functional or integration test environment rather than a unit test environment. Test the system overall focusing on specific areas.

Of course new development will benefit from TDD. As new features are added, refactoring for TDD might help to test the new development, while also allowing development of new unit test for the legacy functions.

Improve this answer
2
  • 4
    We did TDD for about one and a half year, all pretty passionate about it. Yet comparing the TDD projects with earlier ones done without TDD (but not without tests), I wouldn't say that they're actually more stable, or have better designed code. Maybe it's our team: We pair and review a lot, our code quality has always been pretty good.
    – futlib
    Commented Jul 20, 2012 at 4:16
  • 1
    The more I think about it, the more do I think that TDD just didn't fit the technology of that particular project very well: Flex/Swiz. There is a lot of events and bindings and injection going on that make interactions between objects complicated and almost impossible to unit test. Decoupling those objects doesn't make it better, because they work correct in the first place.
    – futlib
    Commented Jul 20, 2012 at 4:17
2

I haven't done TDD in C++ so i can't comment on that, but you're supposed to test the expected behaviour of your code. While the implementation can change, the behaviour should (usually?) stay the same. In Java\C# centric world, that would mean you only test the public methods, writing tests for the expected behaviour and doing that before implementation (which is usually better said than done :)).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.