142

In my current project (a game, in C++), I decided that I would use Test Driven Development 100% during development.

In terms of code quality, this has been great. My code has never been so well designed or so bug-free. I don't cringe when viewing code I wrote a year ago at the start of the project, and I have gained a much better sense for how to structure things, not only to be more easily testable, but to be simpler to implement and use.

However... it has been a year since I started the project. Granted, I can only work on it in my spare time, but TDD is still slowing me down considerably compared to what I'm used to. I read that the slower development speed gets better over time, and I definitely do think up tests a lot more easily than I used to, but I've been at it for a year now and I'm still working at a snail's pace.

Each time I think about the next step that needs work, I have to stop every time and think about how I would write a test for it, to allow me to write the actual code. I'll sometimes get stuck for hours, knowing exactly what code I want to write, but not knowing how to break it down finely enough to fully cover it with tests. Other times, I'll quickly think up a dozen tests, and spend an hour writing tests to cover a tiny piece of real code that would have otherwise taken a few minutes to write.

Or, after finishing the 50th test to cover a particular entity in the game and all aspects of it's creation and usage, I look at my to-do list and see the next entity to be coded, and cringe in horror at the thought of writing another 50 similar tests to get it implemented.

It's gotten to the point that, looking over the progress of the last year, I'm considering abandoning TDD for the sake of "getting the damn project finished". However, giving up the code quality that came with it is not something I'm looking forward to. I'm afraid that if I stop writing tests, then I'll slip out of the habit of making the code so modular and testable.

Am I perhaps doing something wrong to still be so slow at this? Are there alternatives that speed up productivity without completely losing the benefits? TAD? Less test coverage? How do other people survive TDD without killing all productivity and motivation?

Improve this question
6
  • @Nairou: You could always try "getting the project finished"! Make a branch now. Just write the code in there. But limit what you do, either by time or number of game entities and see if you went faster. You can then ignore that branch, go back to trunk and TDD from there and see what the difference is.
    – quamrana
    Jun 22, 2011 at 11:00
  • 12
    To me, writing tests too early is like optimizing too early. You might be working hard on testing code you will remove in the future anyway.
    – LennyProgrammers
    Jun 28, 2011 at 13:49
  • I'm a little concerned you are spending hours thinking of a way to design your code so that it is more testable. Testability is a likely attribute of a well-factored design but it should not be the overriding goal of the design.
    – Jeremy
    Jun 28, 2011 at 14:17
  • 3
    Back when I was learning, we had a trick for when we had to provide design documents. We wrote the code first, then wrote the documents to describe the code. Maybe you need to learn a moderate amount of that pragmatism for your TDD. If you already have a plan in mind, maybe it's better to get most of that into code before writing the tests. Whatever idealism suggests, sometimes it's better to do what you're already ready to do, rather than distract yourself with something else then come back when you're not fresh any more.
    – user8709
    Jul 8, 2011 at 0:22
  • 4
    I'm going to go against the popular opinion and say that TDD may not always be the right choice if you're making games. Since someone at gamedev.stackexchange already answered this question spectacularly, I'll just link this here.
    – TtT23
    Oct 28, 2013 at 6:39

12 Answers 12

Reset to default
84

Let me begin by thanking you to share your experience and voicing out your concerns... which I have to say are not uncommon.

  • Time/Productivity: Writing tests is slower than not writing tests. If you scope it to that, I'd agree. However if you run a parallel effort where you apply a non-TDD approach, chances are that the time you spend break-detect-debug-and-fix existing code will put you in the net negative. For me, TDD is the fastest I can go without compromising on my code-confidence. If you find things in your method, that are not adding value, eliminate them.
  • Number of tests: If you code up N things, you need to test N things. to paraphrase one of Kent Beck's lines "Test only if you would want it to work."
  • Getting stuck for hours: I do too (sometimes and not > 20 mins before I stop the line).. It's just your code telling you that the design needs some work. A test is just another client for your SUT class. If a test is finding it difficult to use your type, chances are so will your production clients.
  • Similar tests tedium : This needs some more context for me to write up a counterargument. That said, Stop and think about the similarity. Can you data-drive those tests somehow? Is it possible to write tests against a base-type? Then you just need to run the same set of tests against each derivation. Listen to your tests. Be the right kind of lazy and see if you can figure out a way to avoid tedium.
  • Stopping to think about what you need to do next (the test/spec) isn't a bad thing. On the contrary, it's recommended so that you build "the right thing". Usually if I can't think of how to test it, I usually can't think of the implementation either. It's a good idea to blank out implementation ideas till you get there.. maybe a simpler solution is overshadowed by a YAGNI-ish pre-emptive design.

And that brings me to the final query : How do I get better ? My (or An) answer is Read, Reflect and Practice.

e.g. Of late, I keep tabs on

  • whether my rhythm reflects RG[Ref]RG[Ref]RG[Ref] or is it RRRRGRRef.
  • % time spent in the Red / Compile Error state.
  • Am I stuck in a Red/Broken builds state?
Improve this answer
9
  • 1
    I'm very curious about your comment on data-driving the tests. Do you just mean a single set of tests that process external data (like from a file) rather than retesting similar code? In my game I have multiple entities, and each one is largely different, but there are some common things that get done with them (serializing them over the network, making sure they don't get sent to non-existing players, etc.) So far I haven't found a way to consolidate this, and so have sets of tests for each one that are almost identical, different only in what entity they use and what data it contains.
    – Nairou
    Jun 22, 2011 at 12:25
  • 1
    @Nairoi - Not sure what test runner you are using. I just learned a name for what I wanted to convey. Abstract fixture pattern[goo.gl/dWp3k]. This still requires you to write as many Fixtures as there are concrete SUT types. If you want to be even more concise, look at your runner's docs. e.g. NUnit supports Parameterized and Generic test fixtures (now that I searched for it) goo.gl/c1eEQ Seems like the very thing you need.
    – Gishu
    Jun 22, 2011 at 13:39
  • Interesting, I've never heard of abstract fixtures. I currently use UnitTest++ which has fixtures, but not abstract ones. It's fixtures are very literal, just a way of consolidating test code that you would otherwise repeat in each test, for a given group of tests.
    – Nairou
    Jun 23, 2011 at 2:20
  • @asgeo - can't edit that comment.. the link has picked up a trailing bracket.This should work - goo.gl/dWp3k
    – Gishu
    Jun 24, 2011 at 2:34
  • +1 for 'getting stuck is symptom of the design needs more work', although.. what happens when you get stuck (like me) at the design?
    – lurscher
    Jun 24, 2011 at 21:14
36

You don't need 100% percent test coverage. Be pragmatic.

Improve this answer
5
  • 3
    If you don't have 100% test coverage, you don't have 100% confidence.
    – Christopher Mahan
    Jun 24, 2011 at 23:32
  • 69
    You don't have 100% confidence even with 100% of test coverage. That's testing 101. Tests cannot demonstrate that the code is defect-free; on the contrary, they can only demonstrate that it does contain defects.
    – CesarGon
    Jun 25, 2011 at 0:40
  • 8
    For what it's worth, one of the most passionate TDD advocates, Bob Martin, does not recommend 100% coverage - blog.objectmentor.com/articles/2009/01/31/…. In the manufacturing industry (granted, different from software in many respects), no one goes for 100% confidence because they can spend a fraction of the effort to be 99% confident.
    – Chance
    Jul 4, 2011 at 20:26
  • Also (at least last time I checked the tools we have) code coverage reports relate to whether lines were executed but doesn't include value coverage - e.g. today I had a bug reported where I had all paths through the code executed in tests, but since there was a line like a = x + y and although all lines in the code were executed in tests, the tests only tested for the case where y = 0, so the bug (it should have been a = x - y) was never found in tests.
    – Pete Kirkham
    Feb 26, 2016 at 12:45
  • @Chance - I have read Robert Martin book "Clean coder..." some long name. It said in that book, that it should be asymptotically 100% covered with the tests, which is close to 100%. And the blog link does not work anymore.
    – Darius.V
    Mar 21, 2017 at 19:25
26

TDD is still slowing me down considerably

That's actually false.

Without TDD, you spend a few weeks writing code which mostly works and spend the next year "testing" and fixing many (but not all) of the bugs.

With TDD, you spend a year writing code which actually works. Then you do final integration testing for a few weeks.

The elapsed time is probably going to be the same. The TDD software will be substantially better quality.

Improve this answer
9
  • 6
    So why do I need TDD? "The elapsed time is the same"
    – Peter Long
    Jun 22, 2011 at 2:09
  • 21
    @Peter Long: Code quality. The year "testing" is how we wind up with crap software that mostly works.
    – S.Lott
    Jun 22, 2011 at 2:10
  • 1
    @Peter, you've got to be kidding. The quality of the TDD solution will be far superior.
    – Mark Thomas
    Jun 22, 2011 at 2:12
  • 7
    Why do I need TDD? Kent Beck lists peace of mind as a big one, and it's very compelling to me. I am living in constant fear of breaking stuff when I work on code with no unit tests.
    – Mathias
    Jun 22, 2011 at 5:20
  • 7
    @Peter Long: "The elapsed time is the same"... and at any point during that time you can deliver working code.
    – Frank Shearar
    Jun 24, 2011 at 16:34
21

Or, after finishing the 50th test to cover a particular entity in the game and all aspects of it's creation and usage, I look at my to-do list and see the next entity to be coded, and cringe in horror at the thought of writing another 50 similar tests to get it implemented.

This makes me wonder how much you're following the "Refactor" step of TDD.

When all your tests are passing, this is time for you to refactor the code and remove duplication. While people usually remember this, sometimes they forget that this is also time to refactor their tests as well, to remove duplication and simplify things.

If you have two entities that merge into one to enable code re-use, consider merging their tests as well. You really only need to test incremental differences in your code. If you're not regularly performing maintenance on your tests, they can quickly become unwieldy.

A couple of philosophical points about TDD that might be helpful:

  • When you can't figure out how to write a test, despite extensive experience writing tests, then that's definitely a code smell. Your code is somehow lacking in modularity, which makes writing small, simple tests difficult.
  • Spiking out a bit of code is perfectly acceptable when using TDD. Write the code you want to, to get an idea of what it looks like, then delete the code and start with tests.
  • I see practicing extremely strict TDD as a form of exercise. When you first start out, definitely write a test first every single time, and write the simplest code to make the test pass before moving on. However, this is not necessary once you become more comfortable with the practice. I don't have a unit test for every single possible code path I write, but through experience I'm able to choose what needs to be tested with a unit test, and what can be covered by functional or integration testing instead. If you've been practicing TDD in a strict fashion for a year, I would imagine you're close to this point as well.

EDIT: On the topic of philosophy of unit testing, I think this might be interesting for you to read: The Way of Testivus

And a more practical, if not necessarily very helpful, point:

  • You mention C++ as your development language. I've practiced TDD extensively in Java, using excellent libraries like JUnit and Mockito. However, I've found TDD in C++ to be very frustrating due to the lack of libraries (in particular, mocking frameworks) available. While this point doesn't help you in your current situation much, I hope you take it into consideration before ditching TDD altogether.
Improve this answer
7
  • 6
    Refactoring tests is dangerous. Nobody seems to talk about this, but it is. I certainly don't have unit tests to test my unit tests. When you refactor to reduce duplication, you typically increase complexity (because your code becomes more general). That means there's more likely to be a bug in your tests.
    – Scott Whitlock
    Jun 24, 2011 at 1:17
  • 2
    I disagree that refactoring tests is dangerous. You are only refactoring when everything is passing, so if you do a refactoring and everything is still green, then you're just fine. If you're thinking you need to write tests for your tests, I feel like that's an indicator that you need to write simpler tests.
    – jaustin
    Jun 24, 2011 at 1:56
  • 1
    C++ is hard to unit test (the language doesn't easily do things that make mocks easy). I've noticed that functions which are "functions" (only operate on arguments, results come out as return values/params) are much easier to test than "procedures" (return void, no args). I've found that it can actually be easier to unit test well-crafted modular C code than C++ code. You don't have to write in C, but you can follow the modular C example. It sounds completely insane, but I've put unit tests on "bad C" where everything was global and it was super easy- all the state is always available!.
    – anon
    Jun 24, 2011 at 2:58
  • 2
    I think this is very true. I do a lot of RedGreenRedGreenRedGreen (or, more often, RedRedRedGreenGreenGreen), but I rarely refactor. My tests certainly have never been refactored, as I always felt it would waste even more time not coding. But I can see how it might be the cause of the issues I'm now facing. Time for me to seriously look into doing some refactoring and consolidation.
    – Nairou
    Jun 25, 2011 at 16:25
  • 1
    Google C++ mocking framework (integrated with google C++ test fw) - very, very powerfull mocking library - flexible, feature rich - quite comparable with any other mocking framework out there.
    – ratkok
    Jun 25, 2011 at 18:25
10

Very interesting question.

What's important to note, is that C++ is not very easily testable, and gaming, in general, is also a very bad candidate for TDD. You can't test if OpenGL/DirectX draws triangle red with driver X, and yellow with driver Y easily. If the bump map normal vector is not flipped after shader transforms. Nor you can test clipping issues over driver versions with different precisions, and so on. Undefined drawing behavior because of incorrect calls can also be tested only with accurate code review and SDK at hand. Sound is also a bad candidate. Multithreading, which again is quite important for games, is pretty much useless to unit-test. So it's tough.

Basically gaming is a lot of GUI, sound and threads. GUI, even with standard components you can send WM_ to, is tough to unit-test.

So what you can test, is model loading classes, texture loading classes, matrix libraries and some-such, which is not a lot of code, and, quite often, not very reusable, if it's only your first project. Also, they are packed into proprietary formats, so it's not quite likely that 3rd party input can differ much, unless you release modding tools etc.

Then again, I'm not TDD guru or evangelist, so take all that with a grain of salt.

I would probably write some tests for main core components (for example matrix library, image library). Add a bunch of abort() on unexpected inputs in every function. And most importantly, concentrate on resistant/resilient code that doesn't break easily.

Concerning off by one errors, clever use of C++, RAII and a good design goes a long way to prevent them.

Basically you have a lot to do just to cover the basics if you want to release the game. I'm not sure if TDD will help.

Improve this answer
3
  • 4
    +1 I really like the concept of TDD and use it wherever I can, but you raise a very important point that TDD advocates are curiously silent on. There are lots of types of programming as you have pointed out for which writing meaningful unit tests is extremely hard if not impossible. Use TDD where it makes sense, but some types of code are better developed and tested in other ways.
    – Mark Heath
    Jul 8, 2011 at 9:58
  • @Mark: yup, no-one seems to care about integration tests nowadays, thinking that becuase they've got an automated test suite, everything wil magically work when put together and tried out with real data.
    – gbjbaanb
    Jul 8, 2011 at 10:24
  • Agree with this. Thanks for a pragmatic answer that doesn't dogmatically prescribe TDD as the answer to everything, instead of what it is, which is just another tool in the developer toolkit.
    – j b
    Mar 3, 2019 at 10:43
6

How do other people survive TDD without killing all productivity and motivation?

This is completely different from my experiences. You are either amazingly intelligent and write code without bugs, (eg. off by one errors) or you don't realize your code has bugs which prevent your program working, and so aren't actually finished.

TDD is about having the humility to know that you (and me!) make mistakes.

For me the time writing unittests is more than saved in reduced debugging time for projects that are done using TDD from the beginning.

If you don't make mistakes then maybe TDD isn't as important for you as it is for me!

Improve this answer
2
  • Well, you have bugs in your TDD code as well ;)
    – Coder
    Jun 28, 2011 at 13:53
  • That true! but they do tend to be a different type of bug, if TDD is done properly. I guess saying code has to be 100% bug free to be finished isn't right. Although if one defines a bug as a deviation from the unit test defined behavior, then I guess it is bug free :)
    – Tom
    Jul 4, 2011 at 20:02
6

I agree with the other answers, but I also want to add a very important point: Refactoring costs!!

With well written unit tests, you can safely do re-writes of your code. First, well written unit tests provide excellent documentation of the intent of your code. Second, any unfortunate side effects of your refactoring will be caught in the existing test suite. Thus, you have guaranteed that the assumptions of your old code are true for your new code as well.

Improve this answer
4

I've only a few remarks:

  1. It seems you're trying to test everything. You probably shouldn't, just the high-risk and edge cases of a particular piece of code / method. I'm pretty sure the 80 / 20 rule applies here: You spend 80% writing tests for the last 20% of your code or cases that aren't covered.

  2. Prioritize. Get into agile software development, and make a list of what you really really really need to do to release in one month time. Then release, just like that. This'll make you think about the priority of features. Yeah, it'd be cool if your character could do a backflip, but does it have business value?

TDD is good, but only if you don't aim for 100% test coverage, and it's not if it keeps you from producing actual business value (i.e. features, things that add something to your game).

Improve this answer
2

Yes, writing tests and code might take longer than just writing code - but writing code and associated unit tests (using TDD) is much more predictable than writing code and then debugging it.

Debugging is almost eliminated when using TDD - which makes all development process much more predictable and in the end - arguably faster.

Constant refactoring - it is impossible to do any serious refactoring without comprehensive unit test suite. The most efficient way to build that unit testing based safety net is during TDD. Well refactored code significantly improves overall productivity of the designer/team who maintains the code.

Improve this answer
2

If good practice would take less upfront effort than bad practice, everyone would already be doing it without needing to enforce it.

It is the very nature of good practice that it includes additional upfront effort. It pays back dividends in the long run, but without clear enforcement developers are liable to use shortcuts that net them quick wins but long term drains on maintainability.

That being said, I don't like dogma either. Pragmatism is very much relevant here, but it is sometimes difficult to find that pragmatic balance. There are two dogmas here that I would advise not following:

  • 100% code coverage
  • Tests musts be written before the implementation

Instead, I would pragmatically suggest that:

  • Ensure code coverage for the non-trivial important bits
  • Don't implement something without also backing it by tests - but the order of these two steps is up to you.

This will ease the tension that purism tends to create. Note that both of these are phrased with some subjectivity baked in. What is considered important? Should you write tests first or second?

Both are reasonable topics that experts can have differing opinions on, and I'm not going to tell you I know your codebase better than you. I would argue that if you stay within the lines of the advice that I set out here, that you're not going to run foul of TDD's spirit, if not its dogma.

Improve this answer
1
  • 1
    Releasing you from dogmas is how you make any development technique to be productive. Funny fact, TDD is, essentially, how Kent Beck approached software development. It's his way to develop. The same K.B said, that's how he works but it's not supposed to be the way that best works for you. Basically, you can take it as is or take whatever works for you and fits in your development style. The second is ergonomic development and how we adopt techniques in a productive way. After all, We are not Sr Kent Beck.
    – Laiv
    May 18 at 7:04
0

Consider narrowing the scope of your game and get it where someone can play it or you release it. Maintain your testing standards without having to wait too long to release your game might be a middle-ground to keep you motivated. The feedback from your users may provide benefits in the long-term and your testing allows you to feel comfortable with the additions and changes.

Improve this answer
0

Here is a PluralSight article on evidence-base conclusions around TDD.

Some results looked at by the article are summarised below. The columns give the research source, and the rows show the outcome being measured.

  • Tick = improves the outcome
  • Circle with minus sign = 'inconclusive'
  • Cross = worsens the outcome
  • The controlled experiment shows increased productivity when using TDD
  • The pilot study is inconclusive on productivity when using TDD
  • Semi-industry research shows a reduction in productivity when using TDD
  • Industry research shows a short term reduction in productivity when using TDD, but a gain in the long term

enter image description here

Also see - https://theqalead.com/general/statistics-studies-benefits-test-driven-development/

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.