I am investigating techniques and strategies for scaling our growing number of integration tests on our current product, so that they can (humanly) remain part of our development, and CI process.
At about 200+ integration tests we are already hitting the 1hr mark to complete a full test run (on aa desktop dev machine), and this is negatively affecting a developer's ability to tolerate running the whole suite as part of the routine push processes. Which is affecting motivation to be disciplined about creating them well. We integration test only key scenrios front to back, and we use an environment that mirrors production, that is built from scratch each test run.
Because of the time it takes to run, it is making for a terrible feedback loop and many wasted cycles waiting on machines to finish test runs, no matter how focused the test runs are. Nevermind the more expensive negative impact on flow and progress, sanity and sustainability.
We expect to have 10x fold more integration tests before this product begins to slow down (no idea really, but it does not feel like we are even getting started in terms of features yet). We have to resaonably expect to be in the few hundreds or or couple thousands of integration tests, I reckon at some point.
To be clear, to try to prevent this becoming a discussion on unit testing versus integration testing, (which should never be traded). We are doing both unit testing with TDD AND integration testing in this product. In fact, we do integration testing at the various layers in the services architecture that we have, where it makes sense to us, as we need to to verify where we introduce breaking changes when changing the patterns in our architecture to the other areas of the system.
A little about our tech stack. We are currently testing on a (CPU and memory intensive) emulation environment to run our tests from end to end. Which is composed of Azure REST web services fronting a noSql backend (ATS). We are simulating our production environment by running in the Azure desktop Emulator + IISExpress. We are limited to one emulator and one local backend repository per dev machine.
We also have a cloud-based CI as well, which runs the same test in the same emulated environment, and the test runs are taking twice as long (2hrs+) in the cloud with our current CI provider. We have reached the limits of the cloud CI providers SLA in terms of hardware performance, and exceeded their allowance on test run time. To be fair to them, their specs are not bad, but half as good as an inhouse grunty desktop machine clearly.
We are using a testing strategy of rebuilding our data store for each logical group of tests, and preloading with test data. While comprehensively insuring data integrity, this adds 5-15% impact on each test. So we think there is little to be gained optimizing that testing strategy at this point in the product development.
The long and the short of it is that: whilst we could optimize the throughput of each test (even if by as much as 30%-50% each), we still wont scale effectively in the near future with several hundred tests. 1hr now is even still far in excess of humanly tolerable, we need an order of magnitude-ish improvement in the overall process to make it sustainable.
So, I am investigating what techniques and strategies we can employ to drastically reduce testing time.
- Writing less tests is not an option. Lets please not debate that one in this thread.
- Using faster hardware is definately an option, although very expensive.
- Running groups of tests/scenarios on separate hardware in parrallel is also definately a preferred option.
- Creating grouping of tests around features and scenarios under development is plausible, but ultimately not reliable in proving full coverage or confidence that the system is not affected by a change.
- Running in a cloud-scaled staging environment instead of running in the desktop emulator is technically possible, although we start adding deployment times to test runs (~20mins each at the start of the test run to deploy the stuff).
- Dividing the components of the system into independent logial pieces is plausible to a degree but we expect limited mileage on that, since the interractions between components is expected to increase with time. (i.e. a change is ine is likely to affect others iin unexpected ways - as often happens when a system is developed incrementally)
I wanted to see what strategies (and tools) others are using in this space.
(I have to believe others may be seeing this kind of difficulty using certain technology sets.))
[Update: 12/16/2016: We ended up investing more in CI parallel testing, for a discussion of the outcome: https://web.archive.org/web/20200114101702/http://www.mindkin.co.nz/blog/2015/12/16/16-jobs ]