Over the past few weeks I've been mulling and researching how to fill a gap in our testing methodology. In simplified terms unit tests are too small and traditional integration tests are too big.

A frequent scenario comes up where A and B both use component C. However A and B have slightly different requirements for, and make slightly different assumptions about C. If I'm the developer of A how and where do I test my assumptions about C?

Obviously unit testing A with mocked assumptions about C is fine for testing A in isolation, but it doesn't test the assumptions themselves.

Another possibility is to add unit tests for C. However this isn't ideal because, while A is in development, altering the tests for C with evolving assumptions from A will excessively be clumsy. Indeed As developer might not even have adequate access to the unit tests of C (e.g an external library).

To frame this with a more concrete example: Assume that this is a node application. A, and B depend on C to read a file (among other things) and store the file contents in object passed to C. At first all files that C handles are small and can be read synchronously without significant blocking. However the developer of B realizes that his files are getting huge and needs to switch C to a async read. This results in a sporadic synchronization bug in A, which is still assuming C is reading files synchronously.

This is the type of bug that is notoriously difficult to track down from full integration tests, and may not be caught in integration tests at all. It is also not caught by As unit tests because the As assumptions are mocked. However it could easily be caught by a "mini" integration test that exercises just A and C.

I've only found a few references to this type of testing. Integration in the Small, Component Integration Testing, Unit Integration Testing. It also relates somewhat to BDD testing direction rather than formal TDD unit testing.

How do I fill this testing gap? Specifically -- where do I put such tests? How do I mock the inputs of A and C for "mini" integration tests? And how much effort should be put into separating testing concerns between these tests and unit tests? Or is there a better way to fill the testing gap?

  • 1
    have you considered versioning of modules A-C and to use some form of dependency management? – miraculixx Jul 6 '14 at 17:50
  • 2
    recommended reading: Why is asking a question on “best practice” a bad thing? – gnat Jul 6 '14 at 18:02
  • 1
    @gnat Thanks for the tip. I made the question less vague. – mjhm Jul 6 '14 at 21:36
  • @miraclixx Thanks for your suggestion. Could you elaborate? If you mean something like blog.nodejitsu.com/package-dependencies-done-right -- I think this solves a different problem than I'm asking about. The components I'm referring to are generally too small to independently version as a node module -- for example a Model or Controller component file. Additionally versioning only gives hints about safety and sources of failures, rather than explicit testing for specific problems. – mjhm Jul 6 '14 at 21:58

It appears to me that you have a fundamental problem with your components.

C should do what C needs to do, and be tested, documented and designed to do just that. When you have a situation where C is designed to "do what B wants" you have an abusive relationship, one that becomes very clear when A arrives and wants C to do something slightly different.

What you should not be doing is unit testing C in the context of A, and especially not A in the context of C - you test A independently, and you provide the results of a mocked C to A. If the real-world version of C does not provide those same results, then you have a bug or design flaw in C that will be caught when you perform your big integration tests. Unit testing has always been this way - you cannot test one unit by testing another unit at the same time. Unit testing just isn't designed to do that.

Integration tests do not have to be "the entire program", though they frequently as set up that way. They can be a test rig that runs A and C together without running the rest of the program (or as little of it as you can get away with). At this point I can't advise further as it depends what these components do and how they interact with the rest of your program but its usually the case that you can write a test rig that provides test coverage of both components. Whether it is worth the effort to do that, or whether it is more efficient to integration test the whole program as one (even if you run a subset of the integration tests) is something only you can answer. Most integration tests are comprised of many sections so hopefully you should be able to run just those relevant to these 2 components (and if not, get your integration test suite so it does work in many pieces).

  • Yeah, that's kinda what I'm thinking. This is beyond the scope and purpose of unit testing. Unfortunately I don't live in a software world where dependent components are perfectly designed, tested, and documented. And what little integration testing we have is generally end-to-end and handled by QA specialists, rather than the source developers. As you might guess there are management and organizational issues in the mix. – mjhm Jul 7 '14 at 2:20
  • I guess you will have to add your own integration tests, but call them unit tests, "we're unit testing the customer login module" as you run up cucumber or selenium. – gbjbaanb Jul 7 '14 at 7:46

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.