Should integration test be included in continuous integration (CI)?

Question

Assume that we are developing a web application, and Hudson does typical jobs such as compile, unit test and static code analysis.

But the tricky part is: Hudson deploys and starts up application server to do integration tests, once the previous jobs done.

That means some difficult things, such as database connection, 3rd part application connection, socket port listening, environment variables, server start up failure handing, etc. We have to set up and tear down these things correctly each time, it’s hard thing. Worse, the integration tests can break the integration test easily.

Do you think if integration test should be included in continues integration (CI)? Can it be manually? Or simplify integration test?

Answer 1

the name Continuous Integration says a lot. What better place to do integration testing than where you're integrating already?
Of course it should not be the only place those tests take place, when developing you should try to ensure that you don't break things after all, not just that your changes work in isolation.
In the end, it's the responsibility of every team member that things don't break, trying to lay blame and rigidly define people or stages to which testing is restricted are counter productive.

Answer 2

To first answer your question: Yes, they are definitely a part of Continuous Integration if you're asking me. But I think we need to clarify what integration tests are.

Martin Fowler was talking about continuous delivery as a way to automate the complete build process to deploy rapidly. This requires developers to get rapid feedback provided by the Continuous Integration process. So he defines stages that the build should go through:

1. a commit build
2. thorough testing
3. deployment

The commit build should not take longer than 10 Minutes he states, because of the rapid feedback for developers.

Here is how I see things: In the first step, fetch the latest commit and build it. If this is successful, you run your unit tests to find out if your classes/class groups are working as defined and expected.

When this is successful you get to the integration test part. Here you test interaction of the just successfully tested units. This involves feeding the units with input and watching their state/interaction/output. Remember that we are still in the commit build, so we want this to be fast as well. So interactions with the file system, a database, network peers and the like have to be stubbed for rapid execution. Martin Fowler also hints the use of in-memory-databases if you require them, just to keep execution on the CI server fast.

After you made sure that the units are working and interacting as required, you usually want to find out about test coverage (just testing a small subsystem usually just isn't enough) and make the tested artifacts available for functional testing/QA/deployment (read: thorough testing) if you think you tests cover enough of your program. Just then, you provision a test environment that mirrors the production environment you are targeting and run tests that involve a real database, real files, real network peers, etc.

In the end, integration tests are about code changes. You want to make sure that the changes you made are not breaking the current system, meaning they integrate well. To find out if they are, you need to make sure that they behave correctly in themselves, then if they interact correctly with their dependencies, and if they were tested at all. You can be quiet confident about your system after you passed all those tests.

If later stages find any problems with your program (like when your database returns a certain value, your network connection will halt) you should try to get these tests stubbed out in the integration tests. The commit build most likely is faster than QA ;)

Answer 3

Do you think if integration test should be included in continues integration (CI)?

If you have integration tests that are passing, and do not require someone to actually stand there and press buttons, then yes - you should add them into CI system.

But, since the integration tests can take very long to execute, you should limit how often they are run. They can be executed during a night, when the CI server is idle.