Where should I draw the line between unit tests and integration tests? Should they be separate?

Question

I have a small MVC framework I've been working on. It's code base definitely isn't big, but it's not longer just a couple of classes. I finally decided to take the plunge and start writing tests for it(yes, I know I should've been doing that all along, but it's API was super unstable up until now)

Anyway, my plan is to make it extremely easy to test, including integration tests. An example integration test would go something along these lines:

Fake HTTP request object -> MVC framework -> HTTP response object -> check the response is correct

Because this is all doable without any state or special tools(browser automation etc), I could actually do this with ease with regular unit test frameworks(I use NUnit).

Now the big question. Where exactly should I draw the line between unit tests and integration tests? Should I only test one class at a time(as much as possible) with unit tests? Also, should integration tests be placed in the same testing project as my unit testing project?

Answer 1

Integration vs. unit tests

You should keep your unit tests and your integration tests completely separated. Your unit tests should test one thing and one thing only and in complete isolation of the rest of your system. A unit is loosely defined but it usually boils down to a method or a function.

It makes sense to have tests for each unit so you know their algorithms are implemented correctly and you immediately know what went wrong where, if the implementation is flawed.

Since you test in complete isolation while unit-testing you use stub and mock objects to behave like the rest of your application. This is where integration tests come in. Testing all units in isolation is great but you do need to know if the units are actually working together.

This means knowing if a model is actually stored in a database or if a warning is really issued after algorithm X fails.

Test driven development

Taking it a step back and looking at Test Driven Development (TDD) there are several things to take into account.

1. You write your unit test before you actually write the code that makes it pass.
2. You make the test pass, write just enough code to accomplish this.
3. Now that the test passes it is time to take a step back. Is there anything to refactor with this new functionality in place? You can do this safely since everything is covered by tests.

Integration first vs Integration last

Integration tests fit into this TDD cycle in one of two ways. I know of people who like to write them beforehand. They call an integration test an end-to-end test and define an end to end test as a test that completely tests the whole path of a usecase (think of setting up an application, bootstrapping it, going to a controller, executing it, checking for the result, output, etc...). Then they start out with their first unit test, make it pass, add a second, make it pass, etc... Slowly more and more parts of the integration test pass as well until the feature is finished.

The other style is building a feature unit test by unit test and adding the integration tests deemed necessary afterwards. The big difference between these two is that in the case of integration test first you're forced to think of the design of an application. This kind of disagrees with the premise that TDD is about application design as much as about testing.

Practicalities

At my job we have all our tests in the same project. There are different groups however. The continuous integration tool runs what are marked as unit tests first. Only if those succeed are the slower (because they make real requests, use real databases, etc) integration tests executed as well.

We usually use one test file for one class by the way.

Suggested reading

1. Growing object-oriented software, guided by tests This book is an extremely good example of the integration test first methodology
2. The art of unit testing, with examples in dot.net On unit testing, with examples in dot.net :D Very good book on principles behind unit-testing.
3. Robert C. Martin on TDD (Free articles): Do read the first two articles he linked there as well.

Answer 2

What is important in any test strategy, is Test Coverage - ie being able to show that all functionality is being tested.

I general, and unless you have specific requirements to the contrary (eg DO178 Level A, IEC61508 SIL 4 etc) which does not appear to be the case in your situation, then if you can test the full function of class or a module (and demonstrate that you have) at system level, then system level testing is adequate. And so on down. Unit testing is only necessary when you haven't covered the testing further up.

Where exactly should I draw the line between unit tests and integration tests?

Given intergration testing is usually easier, quicker and cheaper, draw the line as far along as you can...

Should I only test one class at a time(as much as possible) with unit tests?

It depends on scope, again... by definition, a unit test is testing a single unit. But if you can fully test a full module in one go, then if you wish, do so. You are in effect fulfilling several unit tests in one hit.

Also, should integration tests be placed in the same testing project as my unit testing project?

No fundamental reason why not... unless the higher level testing is performed by an independent tester, at which point you should only be issuing an executable and minimal instrumentation.

Answer 3

When I have small projects I just pop all the tests into the same project. Since a bigger project would have these split I just ensure that it would be possible to tease them apart should it be necessary.

With unit tests I typically only test one class (SUT) in a file.