What is a good integration test strategy?

Question

I'm getting started on a project wherein I want to have pretty thorough test coverage, and I have the luxury of driving the test strategy. I've settled on a workable plan for unit testing, and I've also settled on using Gherkin to describe features and a port of Cucumber to run the scenarios as end-to-end acceptance tests.

The problem is that I sense that there is a gap in between those two layers. I can test all my units in isolation, and I can test that my features work, but I can think of other things that I'm going to want to test.

I'm also coming from a different project with (poorly-implemented) automated tests that are a very brittle and are a maintenance nightmare, the goal of these tests being to mostly replace manual regression testing. Writing more maintainable tests is a must, but at a higher level I'm not sure that our tests are the right ones.

As an example, given a web application, say there's a form to add an event with start and end dates. As an end-to-end test, we can validate that you can, in fact, add an event. But if your start date is after your end date, then you get an error message, and I wouldn't think that how a trivial user input error is handled belongs in a feature file. On the other hand, there seems to be a pretty strong belief that unit testing the UI isn't worth it; instead, one should do automated integration testing.

So what do I do for this code?

Do I unit test the components related to error messages in general, as well as that this form is going to show them, and skip automating that they actually appear? Do I do the above and then automate that just one error message somewhere shows as intended, and assume the rest will work? Do I try to automate every different potential failure case for every form?

This hits the middle ground of integration testing, of which I am leery. Based on my experience, maintaining a large number of integration tests does not seem worth the value. On the other hand, there is functionality above the unit level and below the feature level that I would ideally like tested, both in the UI and outside of it. And I'm concerned about what kind of confidence automated regression can bring if it isn't hitting everything.

I am perfectly willing to write integration tests, but in the context of when to write tests, what tests to write, and how many to write, what is a good approach to addressing this problem?

Answer 1

It all comes down to being cost-effective and pragmatic. Because you are in charge of testing for this project, be careful not to use every type of testing approach or change things that have worked in the past, or introduce a bunch of new tools or practices just for the sake of doing so. I am not saying that is your plan, but just wanted to state that.

If the GUI contains a lot of logic, which it shouldn't, then it must be tested. Ideally the GUI contains very little logic (or none, but realistically that is challenging to do). If it does contain a lot of logic, see if it is feasible or cheaper to refactor the GUI code versus creating and maintaining a large base of slow GUI tests. If the GUI contains little to no logic, then you can maybe get away with not testing it at all, or very few very basic tests to make sure the "connections" are all good, as mentioned above. Also since there are only a few, you don't have to worry about their performance and the maintenance overhead isn't as high.

You should integration test the rest of your system (everything exception GUI) at least, though. The system should be designed in such a way that it doesnt require a GUI to drive it, it could be an (automated) command line interface, web service interface, etc.

There should be the highest number of unit tests, then a smaller number of these "core" integration tests, and very few large/ GUI tests.

Answer 2

Do BDD. Write acceptance tests to define the important scenarios of the system. You'll have most of the integration tests that you need, plus valuable documentation and proof that your system does what it's supposed to.

Then add integration tests so that you have good integration coverage. (BobDalgleish's comment is a good way of thinking about integration coverage.) I usually find that missing integration coverage helps me find missing acceptance tests.

Then add unit tests to specify details of behavior and technical correctness. You won't have to unit-test some code at all, because it will already be tested by your acceptance/integration tests. You will think of things to test in other code, and you may even want to re-test paths through that code that are already tested in acceptance/integration tests so that the unit tests make sense without reading the integration tests.

Answer 3

I tend to think that you should start from the bottom and work up, in terms of testing. Unit testing first: the smallest, most granular pieces. Without them, the rest of the tests don't really have much of a foundation under them.

Integration testing would come next. I would handle integration testing before UI/UAT. These can be written almost the same as unit tests, but categorized differently (in such a way that your test running can do quick-feedback unit tests without integration tests, or vice versa, or both together).

Typically unit tests would be fast and run whenever a commit is checked in. Integration tests would be a pre-release type of thing. UI/UAT comes last.

To pick on your specific "start date must not be later than end date" example, I would think this should definitely be a unit test. It doesn't require any other integration components to test it. However, "make sure the entry writes to the database" for example, would be an integration test.

I think, by default, the domain/business logic layer(s) of a project get far more unit test coverage than, say, an MVC layer. The most important unit tests are done against the business login, IMO. Integration testing is really where you wire together the MVC layer and, say, the domain/BL layer and data access layer to make sure all is working properly.

As such, it's important that the basic architecture of the system is correct. The business object should be validating, for example, that Start Date is not later than End Date. This then becomes very natural for unit testing, given that unit tests really tend to hit the domain layer hard, but not as much on the MVC layer. (of course, the MVC layer could/should verify this before it really gets to the domain, but that's a nicety, not a necessity).

So, with all that said, I'll typically do something like the following:

Project.Domain
Project.Domain.Tests.Unit
Project.WebApp
Project.WebApp.Tests.Unit
Project.WebApp.Tests.Integration

The integration tests are handle much the same, but are given the categorization that your test runners can use to run only unit, or unit and integration together, etc.

Some stuff in the WebApp project might still want to be unit tested: controllers to make sure they're returing JSON correctly, etc, but generally speaking the unit testing tends to get the biggest bang for the buck on the domain level, and respectively integration to presentation/data layers.

And once all that's in place, wrapping everything up with some UAT/UI testing is good too!