How much optimisation should one put in for slow integration test setups

Question

I have some very slow integration tests that use Selenium and require a lot of database setup. The setup and tear down times are in the order of tens of seconds while the test bodies only take a few seconds each. Since there are hundreds of these tests, this adds up to a lot.

My previous place has an elaborated resource pooling/reuse system that identifies which browser window/database tables are in a "good" state and reuses them instead of re-creating. This cuts setup/tear down time to nearly 0 per test (when no test is failing)!

I have spent a few days doing a simplified version of the Selenium browser reuse code and that has produced very good results. However, the database maintenance code would be much harder (e.g. my old place uses Hibernate to recreate the schema, which I cannot use here). I estimate it would take a week of work at least, but since the time saved is only in CI, I find it difficult to get my manager on board. I believe the faster turn-around would help day-to-day test development as well, but no one seems to care.

I am wondering how much time do people normally spent on slow integration test optimisation, especially in small teams (3-4 people). And how do people justify an involved optimisation like above?

Edit:

A bit more details about the costs: the integration tests run on shared Jenkins boxes overnight. They take 1-2 hours depending on how many tests fail and how busy the shared database box is.

We use them mainly for regression testing. Them passing is a requirement for release sign off; however, management allow minor fixes/hotfixes to be released with only the faster unit and non-Selenium integration tests.

I know two cases in the past year that bugs that would have been caught by the Selenium tests were released. I was hoping I can eliminate this by making the tests to finish within a release timeframe.

Also, a bit of development time can be saved as each manual run of a particular test will be faster. But a back-of-the-envelope calculation shows time spent waiting for tests to run accounts of at most 1% of development time, so probably not worth optimising any further...

Answer 1

Figure out the cost.

Do you have a bunch of old machines that you are using for this that you don't actually need? That's much cheaper than if you are having to buy new machines to make them run faster.

Or is your CI system on github/amazon or somewhere you are directly paying costs for servertime? How much $$$ per year is this?

Does this prevent your team from working (1 vs 5 minutes is a very minor difference, but 30 minutes vs 5 hours is much more meaningful) by stopping your workflow for CI to run?

Once you figure this out, figure out how much it would cost you to fix it - estimate how many hours it would take (and then probably multiply by 2 or 3 as your estimate will almost assuredly be low). If the benefit is better than the cost... it will be easy to sell. If not... well what's the point?

Answer 2

Writing tests, given experience, should be very easy. When tests become difficult to implement or manage, it tends to be an indicator of a bigger problem independent of testing. It's your cue to listen to what the tests are telling you and have a rethink.

You have spent a few days working on your test automation. To what gain? At some point, you will have to make that time back. Are you sure this is going to make your company more productive in the long run?

This is partly the reason why some people refer to integration tests as a Vortex Of Doom. I hate to pre-judge what might be making the test initialisation so slow, but I imagine that there might be a lot of data that needs setting up, indicating a lot of interdependency, and parts of the system that know or are doing too much.

Do you have good quality unit tests in place? Why not invest your time in good quality test-driven design based on focused tests? A good TDD approach will ensure your system is not so complex as to require lots of set-up, as well as providing you with a much leaner set of regression tests.

In answer to your question, I would waste no time on optimising slow integration tests, since you will never get that time back. It might even allow you to dig a deeper hole without realising. I would put that time into refactoring your production code to reduce the need for large amounts of setup, in turn making your production code simpler. Over time, I would steer away from integration tests and towards focused tests.