10

I titled the question jokingly because I'm sure that "it depends," but I have some specific questions.

Working in software that has many deep layers of dependency, my team has become accustomed using mocking fairly extensively to separate each code module from the dependencies beneath it.

Therefore I was surprised that Roy Osherove suggested in this video that you should be using mocking only something like 5% of the time. I'd guess that we're sitting somewhere between 70-90%. I've seen other similar guidance from time to time as well.

I should define what I consider to be two categories of "integration tests" which are so distinct that they really should be given different names: 1) In-process tests which integrate multiple code modules and 2) Out-of-process tests which talk to databases, file systems, web services, etc. It is type #1 that I am concerned with, tests which integrate multiple code modules all in-process.

Much of the community guidance I have read suggests that you should prefer a large number of isolated, fine-grained unit tests and a small number of coarse-grained end-to-end integration tests, because unit tests give you precise feedback on exactly where regressions might have been created, but the coarse tests, which are cumbersome to set up, actually verify more end-to-end functionality of the system.

Given this, it seems necessary to make rather frequent use of mocking to isolate these separate units of code.

Given an object model as follows:

enter image description here

... Also consider that our application's dependency depth goes much deeper than I could fit in this image, so that there are multiple layers N between the 2-4 layer and the 5-13 layer.

If I want to test some simple logical decision being made in unit #1, and if every dependency is constructor-injected into the code module that depends on it such that, say, 2, 3, and 4 are constructor injected into module 1 in the image, I would much rather inject mocks of 2, 3, and 4 into 1.

Otherwise, I would need to construct concrete instances of 2, 3, and 4. This can be more difficult than just some extra typing. Often 2, 3, and 4 will have constructor requirements which can be challenging to satisfy and according to the graph (and according to the reality of our project), I'll need to construct concrete instances of N through 13 to satisfy the constructors of 2, 3, and 4.

This situation becomes more challenging when I need 2, 3, or 4 to behave in some certain way so that I can test the simple logical decision in #1. I might need to understand and "mentally reason about" the entire object graph/tree all at once to get 2, 3, or 4 to behave in the necessary way. It often seems much easier to do myMockOfModule2.Setup(x=> x.GoLeftOrRight()).Returns(new Right()); to test that module 1 responds as expected when module 2 tells it to go right.

If I were to test concrete instances of 2 ... N ... 13 all together, the test setups would be very large and mostly duplicated. Test failures might not do a very good job of pinpointing the locations of regressions failures. Tests would not be independent (another supporting link).

Granted, it is often reasonable to do state-based, rather than interaction-based, testing of the bottom layer, since those modules seldom have any further dependency. But it seems that mocking is almost necessary by definition to isolate any modules above the bottom-most.

Given all of this, can anyone tell me what I might be missing? Is our team overusing mocks? Or is there perhaps some assumption in typical unit testing guidance that the layers of dependency in most applications will be shallow enough that it is indeed reasonable to test all of the code modules integrated together (making our case "special")? Or perhaps differently, is our team not bounding our bounded contexts adequately?

  • It sounds to me like your application could benefit from looser coupling. en.wikipedia.org/wiki/Loose_coupling – Robert Harvey Oct 15 '13 at 19:06
  • 1
    Or is there perhaps some assumption in typical unit testing guidance that the layers of dependency in most applications will be shallow enough that it is indeed reasonable to test all of the code modules integrated together (making our case "special")? <-- This. – Robert Harvey Oct 15 '13 at 19:08
  • Also worth noting: The purpose of regression tests (especially integration testing) is to prove that your software still works, not necessarily to identify where it breaks. You can do that with troubleshooting, fix the problem, and then cover the specific breakage with additional unit tests. – Robert Harvey Oct 15 '13 at 19:10
  • I should have been clearer in the original post, to say that module 1 is only aware of I2, I3, and I4. Module 2 is only aware of I5, I6, and I7. It is only the questionable goal of testing without using mocks that I would provide a concrete 2, 3 and 4 to 1, leading to the challenges I described. Otherwise, we wind up using mocks far more than 5% of the time. – ardave Oct 15 '13 at 19:15
  • I kind of joked about our case being "special" after reading a blog post about many teams defying valuable conventions because they incorrectly felt their situation was "special." But if this is actually our case, this would explain the disparity between some of the community guidance that I've read, and some of my team's actual experiences. (the 5% vs 70-90%) – ardave Oct 15 '13 at 19:17
4

Is our team overusing mocks?

Not at first glance.

If you have 1..13 modules, then each should have its own unit tests, and all (non-trivial, untrusted) dependencies should be replaced by test versions. That can mean mocks, but some people are pedantic with naming, so fakes, shims, null objects... some people refer to all test implementations as "mocks". This might be the source of the confusion about how much is "right".

Personally, I just call all test objects "mocks" because distinguishing between the variety of them isn't often useful. As long as they keep my unit tests fast, isolated and resilient... I don't care what they're named.

  • I would wonder then if there is any general guidance out there for when it is best to test code modules in isolation vs. testing more than one code module, integrated together. It seems like as soon as I integrate any two modules which I could have otherwise isolated, I open myself up to a host of undesirable issues: Lack of pinpointing of regression causes / multiple tests failing for a single regression, excessive test setups, etc. I have my own intuitive sense ("listen to the tests") but this has left me at the 70-90% mock level. – ardave Oct 15 '13 at 22:42
  • 1
    @nono - In my experience, you should test everything in isolation, for the reasons you mention. The only things you don't unit test in isolation are things you can't because they go against some file system or other external resource directly (something has to do that after all). – Telastyn Oct 16 '13 at 0:45
  • After chewing on this for a few days, yours seems like the best possible explanation: If one were to use the strict definition of "mock" as a type of test double used for retrospective interaction/behavior verification, as opposed a dummy test double or a test double which is configured in advance to simulate some certain behavior, then yes, I could see winding up at the 5% level. – ardave Oct 18 '13 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.