15

I'm working on a legacy system (by that I mean it was written without tests). We've tried to test some of the system by writing integration tests that test functionality from the outside.

This gives me some confidence to refactor parts of the code without worry about breaking it. But the problem is these integration tests require a deploy (2+ minutes) and many minutes to run. Also, they are a pain to maintain. They each cover thousands of lines of code and when one of them breaks it can take hour(s) to debug why.

I've been writing lots of unit tests for these functional changes I've been making lately, but before I commit I always do a new deploy and run all integration tests, just to make sure I didn't miss anything. At this point I know my unit tests and some of the integration tests are overlapping what they test.

How do I know when my good unit tests are adequately covering a bad integration test so that I can delete that integration test?

Improve this question

2 Answers 2

Reset to default
18

The easiest metric is to ask, "when was the last time this integration test legitimately failed?" If it has been a long time (there have been a lot of changes) since the integration test failed, then the unit tests are probably doing a good enough job. If the integration test has failed recently, then there was a defect that was not caught by the unit tests.

My preference would generally be to increase the robustness of the integration tests, to the point where they can be reliably run unattended. If they take a long time to run, then run them overnight. They are still valuable even if they are only run occasionally. If these tests are too fragile or require manual intervention, then it may not be worth the time spent in keeping them running, and you may consider discarding those that succeed most often.

Improve this answer
4
  • 3
    +1 for recommending the tests be automated since that leads to the obvious question of "Why kill off an automated test?"
    – user53019
    Jan 30, 2013 at 21:12
  • 1
    Yea, I agree with this. Of course, even that still comes to bite you if you don't have good enough unit test coverage. For instance, we currently have an integration test suite that takes about 6 hours to run... but I don't think a test has EVER been deleted due to my company's focus on compatibility
    – Earlz
    Jan 30, 2013 at 22:13
  • 2
    Perhaps I should start a new question for this, but do you suggest that whenever an integration test legitimately fails I should figure out how to write a unit test that also fails and get them both passing?
    – Daniel Kaplan
    Jan 31, 2013 at 5:49
  • 2
    @tieTYT: Yes, that absolutely sounds like a good idea. Unit tests are good; unit tests for stuff that you know has already broken before are even better.
    – Greg Hewgill
    Jan 31, 2013 at 7:31
8

Units tests are not the holy grail of testing, they are only one tool of many than should be used to test a code base. Therefore no amount of unit tests should ever be considered safe to replace other tests. If you have a bad integration test you should work to make it a good integration test, not replace it with something else, that is like replacing your front door with a perimeter fence and a gate.

Improve this answer
3
  • If this project started from scratch that'd make more sense to me. But my first integration test was "check that you can login" and eventually I made a lot of unit tests that "check that you can login". The integration test crashes all the time if you change the html. This example is completely contrived, but isn't it a good case for removing the integration test?
    – Daniel Kaplan
    Jan 30, 2013 at 22:17
  • 3
    @tieTYT: testing something through the UI leads often to very unstable solution. Nevertheless, testing by the UI is important, it is just sometimes the case that manual tests here produce less effort than trying to automate that test and keep it stable. So when you think this is the case here, you may remove that "integration test" from the list of automated test and add it to your test plan of manual tests.
    – Doc Brown
    Jan 31, 2013 at 12:42
  • 1
    @DanielKaplan would it be possible to update the integration test to be more stable? If it's failing because the html changes sometimes, then possibly test something like "username appears on the page after login" rather than something more specific like "username appears inside this div after login"
    – Jen
    Nov 17, 2016 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.