1

I have some very slow integration tests that use Selenium and require a lot of database setup. The setup and tear down times are in the order of tens of seconds while the test bodies only take a few seconds each. Since there are hundreds of these tests, this adds up to a lot.

My previous place has an elaborated resource pooling/reuse system that identifies which browser window/database tables are in a "good" state and reuses them instead of re-creating. This cuts setup/tear down time to nearly 0 per test (when no test is failing)!

I have spent a few days doing a simplified version of the Selenium browser reuse code and that has produced very good results. However, the database maintenance code would be much harder (e.g. my old place uses Hibernate to recreate the schema, which I cannot use here). I estimate it would take a week of work at least, but since the time saved is only in CI, I find it difficult to get my manager on board. I believe the faster turn-around would help day-to-day test development as well, but no one seems to care.

I am wondering how much time do people normally spent on slow integration test optimisation, especially in small teams (3-4 people). And how do people justify an involved optimisation like above?

Edit:

A bit more details about the costs: the integration tests run on shared Jenkins boxes overnight. They take 1-2 hours depending on how many tests fail and how busy the shared database box is.

We use them mainly for regression testing. Them passing is a requirement for release sign off; however, management allow minor fixes/hotfixes to be released with only the faster unit and non-Selenium integration tests.

I know two cases in the past year that bugs that would have been caught by the Selenium tests were released. I was hoping I can eliminate this by making the tests to finish within a release timeframe.

Also, a bit of development time can be saved as each manual run of a particular test will be faster. But a back-of-the-envelope calculation shows time spent waiting for tests to run accounts of at most 1% of development time, so probably not worth optimising any further...

  • 3
    Doing the optimization takes t_o hours, which has a cost of c_o. Per week, your team loses t_w hours due to slow tests, which costs c_w. This is an investment that would break even after c_o/c_w weeks. If you have nothing more urgent to do but can demonstrate the value of doing this optimization, your manager will likely approve. If your analysis finds the change would have no measurable benefit, then it's good that you manager doesn't let you waste time on this. – amon Mar 14 '16 at 12:24
  • There are many pain points when it comes to developing software, spend the most time alleviating the costly ones, but do yourself a favor and consider the ones that drive you nuts. – JeffO Mar 14 '16 at 13:03
3

Figure out the cost.

Do you have a bunch of old machines that you are using for this that you don't actually need? That's much cheaper than if you are having to buy new machines to make them run faster.

Or is your CI system on github/amazon or somewhere you are directly paying costs for servertime? How much $$$ per year is this?

Does this prevent your team from working (1 vs 5 minutes is a very minor difference, but 30 minutes vs 5 hours is much more meaningful) by stopping your workflow for CI to run?

Once you figure this out, figure out how much it would cost you to fix it - estimate how many hours it would take (and then probably multiply by 2 or 3 as your estimate will almost assuredly be low). If the benefit is better than the cost... it will be easy to sell. If not... well what's the point?

0

Writing tests, given experience, should be very easy. When tests become difficult to implement or manage, it tends to be an indicator of a bigger problem independent of testing. It's your cue to listen to what the tests are telling you and have a rethink.

You have spent a few days working on your test automation. To what gain? At some point, you will have to make that time back. Are you sure this is going to make your company more productive in the long run?

This is partly the reason why some people refer to integration tests as a Vortex Of Doom. I hate to pre-judge what might be making the test initialisation so slow, but I imagine that there might be a lot of data that needs setting up, indicating a lot of interdependency, and parts of the system that know or are doing too much.

Do you have good quality unit tests in place? Why not invest your time in good quality test-driven design based on focused tests? A good TDD approach will ensure your system is not so complex as to require lots of set-up, as well as providing you with a much leaner set of regression tests.

In answer to your question, I would waste no time on optimising slow integration tests, since you will never get that time back. It might even allow you to dig a deeper hole without realising. I would put that time into refactoring your production code to reduce the need for large amounts of setup, in turn making your production code simpler. Over time, I would steer away from integration tests and towards focused tests.

  • 1
    I agree that problems with tests are often just symptoms of deeper design problems, but the idea that integration tests per se are bollocks is a bit too simple to be true. Take for example this page. How would you test that a user can submit answers which are saved to the DB and displayed to all users? An unit test does not cut it; you need both a DB and a full frontend. – amon Mar 14 '16 at 12:48
  • @amon Integration tests are completely necessary; automating them is not. If my manual, integrated test shows a problem, I write another focused test to drive the solution. That problem won't happen again. – Tim Mar 14 '16 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.