How do you scale your integration testing?

Question

I am investigating techniques and strategies for scaling our growing number of integration tests on our current product, so that they can (humanly) remain part of our development, and CI process.

At about 200+ integration tests we are already hitting the 1hr mark to complete a full test run (on aa desktop dev machine), and this is negatively affecting a developer's ability to tolerate running the whole suite as part of the routine push processes. Which is affecting motivation to be disciplined about creating them well. We integration test only key scenrios front to back, and we use an environment that mirrors production, that is built from scratch each test run.

Because of the time it takes to run, it is making for a terrible feedback loop and many wasted cycles waiting on machines to finish test runs, no matter how focused the test runs are. Nevermind the more expensive negative impact on flow and progress, sanity and sustainability.

We expect to have 10x fold more integration tests before this product begins to slow down (no idea really, but it does not feel like we are even getting started in terms of features yet). We have to resaonably expect to be in the few hundreds or or couple thousands of integration tests, I reckon at some point.

To be clear, to try to prevent this becoming a discussion on unit testing versus integration testing, (which should never be traded). We are doing both unit testing with TDD AND integration testing in this product. In fact, we do integration testing at the various layers in the services architecture that we have, where it makes sense to us, as we need to to verify where we introduce breaking changes when changing the patterns in our architecture to the other areas of the system. 

A little about our tech stack. We are currently testing on a (CPU and memory intensive) emulation environment to run our tests from end to end. Which is composed of Azure REST web services fronting a noSql backend (ATS). We are simulating our production environment by running in the Azure desktop Emulator + IISExpress. We are limited to one emulator and one local backend repository per dev machine.

We also have a cloud-based CI as well, which runs the same test in the same emulated environment, and the test runs are taking twice as long (2hrs+) in the cloud with our current CI provider. We have reached the limits of the cloud CI providers SLA in terms of hardware performance, and exceeded their allowance on test run time. To be fair to them, their specs are not bad, but half as good as an inhouse grunty desktop machine clearly.

We are using a testing strategy of rebuilding our data store for each logical group of tests, and preloading with test data. While comprehensively insuring data integrity, this adds 5-15% impact on each test. So we think there is little to be gained optimizing that testing strategy at this point in the product development. 

The long and the short of it is that: whilst we could optimize the throughput of each test (even if by as much as 30%-50% each), we still wont scale effectively in the near future with several hundred tests. 1hr now is even still far in excess of humanly tolerable, we need an order of magnitude-ish improvement in the overall process to make it sustainable.

So, I am investigating what techniques and strategies we can employ to drastically reduce testing time.

• Writing less tests is not an option. Lets please not debate that one in this thread.
• Using faster hardware is definately an option, although very expensive.
• Running groups of tests/scenarios on separate hardware in parrallel is also definately a preferred option.
• Creating grouping of tests around features and scenarios under development is plausible, but ultimately not reliable in proving full coverage or confidence that the system is not affected by a change. 
• Running in a cloud-scaled staging environment instead of running in the desktop emulator is technically possible, although we start adding deployment times to test runs (~20mins each at the start of the test run to deploy the stuff).
• Dividing the components of the system into independent logial pieces is plausible to a degree but we expect limited mileage on that, since the interractions between components is expected to increase with time. (i.e. a change is ine is likely to affect others iin unexpected ways - as often happens when a system is developed incrementally)

I wanted to see what strategies (and tools) others are using in this space.

(I have to believe others may be seeing this kind of difficulty using certain technology sets.))

[Update: 12/16/2016: We ended up investing more in CI parallel testing, for a discussion of the outcome: https://web.archive.org/web/20200114101702/http://www.mindkin.co.nz/blog/2015/12/16/16-jobs ]

Answer 1

I worked at a place that took 5 hours (across 30 machines) to run integration tests. I refactored the codebase and did unit tests instead for the new stuff. The unit tests took 30 seconds (across 1 machine). Oh, and bugs went down too. And development time since we knew exactly what broke with granular tests.

Long story short, you don't. Full integration tests grow exponentially as your codebase grows (more code means more tests and more code means all of the tests take longer to run as there's more "integration" to work through). I would argue that anything in the "hours" range loses most of the benefits of continuous integration since the feedback loop isn't there. Even an order of magnitude improvement isn't enough to get you good - and it's nowhere close to get you scalable.

So I would recommend cutting the integration tests down to the broadest, most vital smoke tests. They can then be run nightly or some less-than-continuous interval, reducing much of your need for performance. Unit tests, which only grow linearly as you add more code (tests increase, per-test runtime does not) are the way to go for scale.

Answer 2

Integration tests will always be long running as they should mimic a real user. For this very reason you shouldn't run them all synchronously!

Given that you are already running stuff in the cloud it seems to me like you are in a prime position to scale your tests over multiple machines.

In the extreme case, spin up one new environment per test and run them all at the same time. Your integration tests will then only take as long as the longest running test.

Answer 3

Cutting down/optimizing the tests seems like the best idea to me, but in case that's not an option, I have an alternative to propose (but requires building some simple proprietary tools).

I faced a similar problem but not in our integration tests (those ran in minutes). Instead it was simply in our builds: large-scale C codebase, would take hours to build.

What I saw as extremely wasteful was the fact that we were rebuilding the entire thing from scratch (about 20,000 source files/compilation units) even if only a few source files changed, and thus spending hours for a change which should only take seconds or minutes at worst.

So we tried incremental linking on our build servers, but that was unreliable. It would sometimes give false negatives and fail to build on some commits, only to then succeed on a full rebuild. Worse, it would sometimes give false positives and report a build success, only for the developer to merge a broken build into the main branch. So we went back to rebuilding everything every time a developer pushed changes from his private branch.

I hated this so much. I would walk into conference rooms with half the developers playing video games and simply because there was little else to do while waiting on builds. I tried to get a productivity edge by multitasking and starting a new branch once I committed so that I could be working on code while waiting for the builds, but when a test or build failed, it became too painful to queue up changes past that point and try to fix everything and stitch it all back.

Side Project While Waiting, Integrate Later

So what I did instead was to make a skeletal framework of the application -- same kind of basic UI and relevant parts of the SDK for me to develop against as a whole separate project. Then I would write independent code against that while waiting for builds, outside the main project. That at least gave me some coding to do so that I could stay somewhat productive, and then I would start integrating that work done completely outside the product into the project later -- side snippets of code. That's one strategy for your developers if they find themselves waiting a lot.

Parsing Source Files Manually to Figure out What to Rebuild/Rerun

Yet I hated how we were wasting so much time to rebuild everything all the time. So I took it upon myself over a couple of weekends to write some code that would actually scan files for changes and rebuild only the relevant projects -- still a full rebuild, no incremental linking, but only of the projects that needs to be rebuilt (whose dependent files, parsed recursively, changed). That was totally reliable and after demonstrating and testing it exhaustively, we were able to use that solution. That cut the average build times from hours to a few minutes since we were only rebuilding the necessary projects (though central SDK changes could still take an hour, but we did that a lot less frequently than localized changes).

The same strategy should be applicable to integration tests. Just recursively parse source files to figure out what files the integration tests depend upon (ex: import in Java, #include in C or C++) on the server side, and the files included/imported from those files and so on, building a full include/import dependency file graph for the system. Unlike build parsing which forms a DAG, the graph should be undirected since it's interested in any file that changed that contains code which could be executed indirectly *. Only re-run the integration test if any of those files in the graph for the integration test of interest have changed. Even for millions of lines of code, it was easy to do this parsing in less than a minute. If you have files other than source code which can affect an integration test, like content files, perhaps you can write metadata into a comment in the source code indicating those dependencies in the integration tests, so that should those external files change, the tests also get re-run.

_{* As an example, if test.c includes foo.h which is also included by foo.c, then a change to either test.c, foo.h, or foo.c should mark the integrated test as needing a fresh run.}

This can take a full day or two to program and test out, especially in the formal environment, but I think should work even for integration tests and it's well-worth it if you have no other choice but to wait in the hours range for builds to finish (either due to the building or testing or packaging process or whatever). That can translate to so many manhours lost in just a matter of months that would dwarf the time it takes to build this kind of proprietary solution, as well as killing the energy of the team and increasing the stress caused by conflicts in bigger merges done less frequently as a result of all the time wasted waiting. It's just bad for the team as a whole when they're spending large portions of their time waiting on things. The easiest and most universal way to optimize is to simply realize the fact that not all changes require everything to be rebuilt/re-run/repackaged on every little change.

Answer 4

Sounds like you have way too many integration tests. Recall Test pyramid. Integration tests belong in the middle.

As an example take a repository with method set(key,object), get(key). This repository is used extensively throughout your code base. All the methods that depend on this repository will be tested with a fake repository. Now you only need two integration tests, one for set and one for get.

Some of those integration tests could probably be converted to unit tests. For example, end to end tests in my view should only test that the site is configured correctly with the correct connection string and correct domains.

Integration tests should test that the ORM, repositories, and queue abstractions are correct. As a rule of thumb, no domain code is needed for integration testing - only abstractions.

Almost everything else can be unit tested with stubbed/mocked/faked/in-mem-implementations for dependencies.

Answer 5

In my experience in an Agile or DevOps environment where continuous delivery pipelines are common, integration testing should be carried out as each module is completed or adjusted. For example, in many continuous delivery pipeline environments, it’s not uncommon to have multiple code deployments per developer per day. Running a quick set of integration tests at the end of each development phase prior to deployment should be a standard practice in this type of environment. For additional information, a great eBook to include in your reading on this subject is A Practical Guide to Testing in DevOps, written by Katrina Clokie.

To efficiently test in this manner, the new component must either be tested against existing completed modules in a dedicated test environment or against Stubs and Drivers. Depending on your needs, it’s generally a good idea to keep a library of Stubs and Drivers for each application module in a folder or library to enable quick repetitive Integration testing use. Keeping Stubs and Drivers organized like this makes it easy to perform iterative changes, keeping them updated and performing optimally to meet your ongoing testing needs.

Another option to consider is a solution originally developed around 2002, called Service Virtualization. This creates a virtual environment, simulating module interaction with existing resources for testing purposes in a complex enterprise DevOps or the Agile environment.

This article can be useful to understand more about how to do integration testing in the enterprise

Answer 6

It sounds like your code base is growing large, and some code management will help. We use Java, so apologies in advance if I assume this.

• A large project needs to be broken down into smaller individual projects that compile to libraries. Java tools like nexus make this easy.
• Every library should implement an interface. This aids stubbing out the library in higher-level tests. This is particularly useful if the library accesses a database or an external datastore (e.g. a mainframe). In such cases, getting the mainframe or database data into a repeatable state will likely be slow, and may be impossible.
• Integration tests for each library can be comprehensive, but only need run when new library source is committed.
• Higher-level integration tests should just call the libraries and assume they are perfect.

The Java shop that I work in uses this approach, and we are seldom held up waiting for integration tests to run.

Answer 7

Have you measured each test to see where the time is being taken? And then, measured the performance of the codebase if there's a particularly slow bit. Is the overall problem one of the tests or the deployment, or both?

Typically you want to reduce the impact of the integration test so that running them on relatively minor changes is minimised. Then you can leave the full test for a 'QA' run which you perform when the branch is promoted to the next level. So you have unit tests for dev branches, run reduced integration tests when merged, and run a full integration test when merged to a release candidate branch.

So this means you do not have to rebuild and re-package and redeploy everything every commit. You can organise your setup, in the dev environment, to perform an as cheaply-as-possible deployment trusting that it will be OK. Instead of spinning up a whole VM, and deploying the entire product, leave the VM with the old version in place and copy new binaries in place, for example (YMMV depending what you have to do).

This overall optimistic approach still requires the full-on test, but that can be performed at a later stage when the time taken is less urgent. (eg you can run the full test once during the night, if there are any issues the dev can resolve them in the morning). This also has the advantage of refreshing the product on the integration rig for the next day's testing - it may get out of date as devs change things, but only by 1 day.

We had a similar problem running a security-based static analysis tool. Full runs would take ages, so we moved running it from the developer commits to an integration commit (ie we had a system where dev said they were finished, it got merged to a 'level 2' branch where more testing was performed, including perf tests. When that was complete it got merged to a QA branch for deployment. The idea is to remove the regular runs that would occur continually to runs that were made nightly - devs would get the results in the morning and they would not affect their development focus until later in their dev cycle).

Answer 8

At some point, a full set of integration tests may take many hours to complete, even on expensive hardware. One of the options is not to run the majority of those tests on every commit, and instead run them every night, or in a continuous batch mode (once per multiple commits).

This, however, creates a new problem - developers don't receive immediate feedback, and broken builds may go unnoticed. To fix this, it is important that they would know that something is broken at all times. Build notification tools like Catlight or TeamCity's tray notifier can be quite useful.

But there will be yet another problem. Even when the developer sees that the build is broken, he might not rush to check it. After all, somebody else may be already checking it, right?

For that reason, those two tools have a "build investigation" feature. It will tell if anyone from the development team is actually checking and fixing the broken build. Developers can volunteer to check the build, and, until that happens, everyone on the team will be annoyed by a red icon near the clock.

Answer 9

Another possible approach to keep in the CI pipeline integration tests (or any kind of verifications, including builds) with long execution times or requiring limited and/or expensive resources is to switch from the traditional CI systems based on post-commit verifications (which are susceptible to congestion) to one based on pre-commit verifications.

Instead of directly committing their changes into the branch developers submit them to a centralized automated verification system which performs the verifications and:

• if successful it automatically commits the changes into the branch
• if unsuccessful it notifies the respective submitters to re-evaluate their changes

Such approach allows combining and testing together multiple submitted changes, potentially increasing the effective CI verification speed many times.

One such example is the Gerrit/Zuul-based gating system used by OpenStack.

Another one is ApartCI (disclaimer - I'm its creator and the founder of the company offering it).