37

Do I understand it right that classical TDD is just about unit tests? Don't understand me wrong: I know the difference between TDD and just unit testing. I am asking whether it is correct to use integration test in TDD workflow.

Currently I work on the project where TDD is surely only about unit tests and there is at least one serious problem with it. The majority of our unit tests are behavioural tests which often become false negative (false red) during refactoring (just because some sequence of dependencies calls changed). The project was created in TDD style to make refactoring simple and suddenly refactoring became a hell. Epic fail!

The most obvious decision now is to make our test not unit but integrational (but still with TDD). So if previously we stubbed/mocked class dependencies, now we won't do it (at least not all of them). As result most of our tests will become state (instead of behavioural). And state tests become false negative much more seldom (because they test the result but not the workflow of execution).

So I would like to know how widespread is an approach of using TDD with integration tests. Is it okay? Would be grateful for any resources on this topic. I have read this article but it is a bit... strange

Update. Here I will clarify what I mean under unit test and integration test. Unit test is a test which stubs/mocks all class dependencies. Integration test has real implementations of dependencies (although it can stub/mock some dependencies if needed).

15
  • "Okay" is that which meets your requirements. "Make your code readable and clear" is "okay" until it becomes necessary to perform some obscure optimization to meet a performance requirement. Commented Aug 9, 2011 at 18:45
  • 2
    So you don't see it as a warning that refactoring has changed the required sequence of calls? Commented Aug 9, 2011 at 19:20
  • 3
    Sounds like somebody doesn't want to deal with the reds. "Red, green, refactor, wait, why is it red? Screw this, TDD sucks." Commented Aug 9, 2011 at 19:55
  • 2
    @Christopher, but sometimes the sequence is arbitrary and doesn't matter and its okay if it changes as a result of a refactor. This suggests to me that the OP is doing something wrong with how he tests. i.e. in cases where the sequence doesn't matter, don't test the sequence. Commented Aug 9, 2011 at 22:35
  • 1
    TDD isn't about unit tests at all. TDD is about design. With TDD your classes and thus your program as a whole has to be designed to be testable. This means low coupling, high cohesion, and all the other benefits you get from SOLID code. The unit tests exist to enforce that, whilst also providing assurance that your code works, and continues to work, when you refactor to further improve design.
    – CraigTP
    Commented Aug 26, 2012 at 10:15

10 Answers 10

20

In TDD, I'm going to test units together when it makes sense. They way I see it, I use mock/stubs for two reasons: the behavior of the system becomes too complex to effectively test with the full implementation, and the full implementation may do things I don't want to happen.

Basically, the more objects are involved in a test, the more difficult the system will be to predict. You've got the predict the behavior of the system in order to write the test. Some object may have complex behavior and it may be difficult induce particular edge cases in one unit. Thus it can be really helpful to mock those objects.

In other cases, the objects have side effects that are undesirable. Suppose you have a CreditCardProcessor. You don't really want to charge credit cards while your tests are running. Other items like drawing graphics or accessing web resources may be in the same category.

When an object has a dependency, how do you decide whether to include the actual object or some sort of mock/stub?

Firstly, if there is any possibility that the behavior of the object will change during development I'd stub it. For example, consider a priority queue class vs a price strategy class. A priority queue will almost certainly always maintain the same behavior. However, your pricing strategy is likely to change a lot. As a result, you don't want other tests to depend on the behavior inside pricing strategy. If they do, you'll just end up breaking other tests needlessly. However, its not really a big deal for priority queue because the behavior should never change.

Secondly, how "fat" is the interface between the objects? If the objects have a very simple interface, then mocking is easy and I'll do it. If the objects have a complex interface then mocking is hard and less likely to be worthwhile. In this case, let's contrast a database connection object and a price strategy. The price strategies interface should be reasonably simple, hopefully just a CalculatePrice(SalesOrderItem) method. Sure, the actual code may do all sorts of things with the SalesOrderItem, but your stub doesn't have to deal with that. On the other hand, a database connection has SQL statements being passed to it which gives it a fairly complex interface. Mocking the database is really hard because you have check all of the queries that are being made and provide correct response. Furthermore, you aren't checking to make sure that the queries are valid (just that they match what you expect), in such cases checking against an actual database makes more sense, that way you actually verify that the queries work and the tests will still pass if you rewrite the queries to give the same results but in a different way.

Thirdly, if an object will be slow I stub it. If you have a database, calls to it will be fairly slow prompting the use of a stub to avoid having to call into it. Similar for web access, etc.

I did just use databases as an example of something you should stub and also not stub. I stub my databases with a sqlite in-memory database which avoids the performance problem but still allows my SQL to be tested. I'm actually using a framework that generates SQL specific to my database for me so that's work.

In your actual case, you state:

The majority of our unit tests are behavioural tests which often became false negative (false red) during refactoring (just because some sequence of dependencies calls changed).

As I understand this, your tests fail because before foo() was called first then bar(). Now bar() is called then foo(). If the order of calling foo() and bar() doesn't matter, your tests shouldn't be checking which called first. Your test should only be verifying that both are called.

5
  • Minor pick on your example - If the pricing strategy code will change that much, you should probably be thinking about a data driven approach, like a DSL.
    – Michael K
    Commented Aug 9, 2011 at 19:11
  • @Michael, I agree. However, the nature of the DSL is likely to change over time. (The variation in pricing strategies knows no bounds.) So I think the example would still apply. Commented Aug 9, 2011 at 19:28
  • I'd be wary of relying too heavily on SQLite. Different DBs have different requirements and what works on one DB may break badly on another DB. Also, the more complex the object, the more mocking is likely to help reduce the brittleness of your unit tests, and increase your ability to understand them later. It sounds to me like you're assuming people will implement their own mocks. I'd suggest in most cases to use a mock object framework, and use a stateless or less stateful mocking style wherever possible. Commented Mar 1, 2013 at 16:35
  • 2
    @MerlynMorgan-Graham, of course you can use a mocking framework. But a mocking framework doesn't alleviate the problems with attempting to mock a database. A typical approach to mocking a database won't test that the queries do what they are expected to do, and will be too fragile against changes in those queries. That's why I advocate testing against an actual mock database, it will verify that the queries do the right thing. Commented Mar 1, 2013 at 20:19
  • 1
    @MerlynMorgan-Graham, yes depending on SQLite is not ideal. There are differences. Those are mostly handled by the database abstraction framework. I also run the tests against the production database engine from time to time as well. Together this works, and actually tests the code far better than if I were to simply mock my database. Commented Mar 1, 2013 at 20:22
33

Refactoring - changing the structure of code without changing the behavior.

Unit test - a test that makes sure a unit of code behaves as expected.

If your unit tests fail because you are changing the structure of your code, then your tests are not testing behavior. They're testing structure. In theory, if your test fails, it should have failed because the tested behavior has changed, regardless of the code structure. So if your tests are failing because of refactoring, then either your tests need to be written better, or your refactoring is bad.

I don't think the problem is with unit testing in general, but rather in the way you are writing your unit tests. If you find it difficult to write your unit tests so that they only test behavior and not structure, then that's an indicator that your code design may need to change. And that's what TDD is all about.

5
  • 1
    Thats very good point! I would upvote twice if I could.
    – Kromster
    Commented Aug 10, 2011 at 5:15
  • I think the intention of this post is correct, but I can see an outcome of the advice to be to test less ("your tests need to be written better"). This would not be ideal. I think the last paragraph deserves a little expansion. Testing internal behaviors is good. But maybe the OP's internal abstractions are poor/brittle and don't allow a refactor without major reshaping of the code. In such a case, throwing out the tests and writing new ones makes perfect sense. The old tests still provided value because they showed the internal code's behavior wasn't well defined/high level enough. Commented Mar 1, 2013 at 17:02
  • Structural changes will cause unit tests to break. A trivial example would be a method name change. There are plenty of other examples though. I have gone through and completely changed the structure of classes/objects by combining them, splitting them, etc. The behavior of the system stayed exactly the same, but the internal structure completely changed. These cases have definitely lead to broken unit tests. Any refactors that occur at a larger granularity than the unit have the potential to break the unit test without changing system behavior.
    – snakehiss
    Commented Mar 5, 2013 at 7:43
  • @dietbuddha Internal structure changes shouldn't cause breakage in your tests, because your tests are focused on testing higher-level inputs and outputs. If you do need to make a structural change that actually affects a test (like renaming a public method), TDD would suggest that you make the change first in the test itself, and then everywhere else. Usually that's pretty trivial to do. If internal changes are causing a significant amount of problems with your tests, then you're probably mocking/stubbing too much.
    – Phil
    Commented Mar 5, 2013 at 17:32
  • @Phil in practice you may refactor a large piece of code with a few tests, but end up breaking a significant number of other tests that rely on that structure. You still go back and fix those tests, but they do get broken get broken.
    – snakehiss
    Commented Mar 6, 2013 at 5:31
10

Unit tests are the easiest to write and maintain when you are writing tests for simple functions. As you move outward from that to more complex methods using intricate data structures, unit tests become increasingly more elaborate and expensive. Mock objects must be employed, and the tests get increasingly fragile.

At some point, you achieve diminishing returns.

However, all is not lost. Practicing TDD (which should really be called Test-Driven Design, not Test-driven Development) forces you to think about your code and write it in ways that make it more unit-testable.

Short answer: The practice of unit testing (of which TDD is a subset) can greatly improve the quality of the code you write. But, like most things in computing, it is not a silver bullet.

1
  • "most things in computing" or "everything in computing"? :P
    – Benjol
    Commented Aug 10, 2011 at 5:32
6

TDD is a mindset.

Building code in a manner that is easily testable is the easiest way to look at this. The type of test should be irrelevant for all but the most extreme cases.

Unit, Integration, and System are all methods of testing and if your development is being approached in a test driven manner then the type of test should be put on the back burner since your code will inevitably move through varying cycles and be tested in varying manners.

0
6

If you want the True Definition of TDD, you should read Kent Beck's Book, Test Driven Development by Example. Different practitioners of TDD will have different levels of testing that they recommend. And different people have different definitions of unit and integration tests*.

If you are defining unit tests as purely isolated, then I don't think you can say that TDD is about unit tests. Some people have taken to the term "developer tests", and stop worrying about what level they exist in. I recommend that you worry less about if your tests are unit tests, and more about if they are useful tests.

Concerning State-based tests versus behavioral/interaction/mock tests, I prefer to use state based tests for items that primarily maintain state, and mock based tests for items that primarily make connections. (In other words, I use state based tests for my Models, and mocking for my Controllers.)

*: Full disclosure; That's my own link. I only wrote it because no one else had yet, and it seemed valuable to point out the terminology land-mine here.

4

At its core, TDD is about feedback. The size of the "System Under Test" merely affects the accuracy of that feedback. The larger the SUT the more likely you are to get false positives. The smaller the SUT, the more likely you are to get those false negatives you describe. I'm sure there's a balance somewhere.

To address this idea of of approaching TDD with "larger" tests, I'd recommend reading Growing Object-Oriented Software, Guided by Tests

This might give you a better idea of what approaches are out there for building an application TDD style from the outside-in. I gather the approach described is pretty widespread and is known as the "London school" of TDD. I certainly learned a lot about setting up common test fixtures and rolling Fakes for those larger integration borders you don't want to cross in every test. The book's large example starts each feature with an acceptance test that spawns smaller tests. Part IV has a lot of great information on test construction - like how to test things like concurrency, synchronization and asynchronous calls, how to make your tests expressive, what the tests are trying to tell you design wise, qualities of a flexible test, etc.

(Caveat: A half-dozen years ago, classical TDD meant state-based vs "mockist." Now apparently classic TDD can use mocks too, and is different than the "London school". Regardless of the terminology, I think there's something to learn outside of classical TDD, even if one doesn't want to mock every object interaction.)

3

I had came to the same conclusion as you did at one point. However, after a while of doing integration tests(in particular web tests using Selenium), I found them painful as well, for different reasons. Later I saw this video titled Integration Tests are a Scam which tipped me once again into the unit testing camp. I am glad you brought up the point of false negatives: that's a term I use as well. The problem is: you can get lots of false negatives with integration tests as well. The trick then, whether you are writing unit tests or integration tests, is to minimize the cost of maintaining the tests while gaining maximum benefit from them - allowing you to add features and refactor code faster by catching errors early. This is very much an art and I can't say I have cracked it.

The main problems with integration tests are

  1. they run slow, really slow, which will cause you not to want to run them
  2. because they test a large number of objects, a failure could come from many different causes; inversely, one problem could cause a large number of tests to fail(false negatives)
  3. when a large number of objects interact, the number of possible code paths gets out of hand. So, it's impractical to test all of them using integration tests.
1

Think about what the different tests show:

A unit test will show that the unit of code does what you decided it should do.

An integration test shows that the units of code (and perhaps other things like databases, file systems, messaging systems, etc.) work together the do what their supposed to do.

A user acceptance test shows that the user likes what all that code and stuff lets them do.

You aren't really done until all three of these things are good. There isn't really a point to writing a method that won't do what it's supposed to--so define that in a unit test first. There isn't a point to writing a bunch of classes that don't work together as they should----so define that in an integration test first. Finally, there's absolutely point developing a system users don't want, so... you figure it out.

1

TDD tests features. Sometimes features are units, sometimes they're integrations or acceptance tests. TDD tests scale.

1

So I would like to know how widespread is an approach of using TDD with integration tests. Is it okay? Would be grateful for any resources on this topic. I have read this article but it is a bit... strange

Acceptance Test Driven Development

The term used for TDD that involves testing the behavior of the system at large rather than a particular unit of code.

I've personally used it to great effect for about 7 years now primarily in web development.

Before you jump in I would caution you against going from one extreme (mocking/stubbing everything outside the unit) to another (not managing external resources at all). You should assess your needs for development. Establish what the trade-offs are and choose whichever one yields the greatest value. Always do the analysis, because "best practices" sometimes are not.

Your Okay, I'm Okay

I wouldn't worry about the practice being "okay". I think you should focus on if it solves your problem, or at least is a marked improvement.

Not the answer you're looking for? Browse other questions tagged or ask your own question.