Automated testing: should I check a sub-function was called, or should I check it's result?

Question

I have a big function that does several things, including some database operations, and calling another smaller function. Something like:

BigFunction()
{
    DB.SomeTable.AddRow(newRow);
    
    SmallFunction();
    
    DB.OtherTable.AddRow(otherRow);
}

SmallFunction();
{
    DB.SmallTable.AddRow(smallRow);
}

SmallFunction is already covered by it's own automated tests.

Now I want to write some automated tests for BigFunction.

When writing the automated tests for BigFunction, should I test that:

1. BigFunction calls SmallFunction (using a mock), or

2. The thing that SmallFunction does, actually happened (in my example: that smallRow is now in SmallTable)?

Some of my team says 1, because SmallFunction already has it's own tests, and so 2 is duplicating an existing test. If the purpose/result of SmallFunction changes, both tests will need to be changed.

Some say 2, because a) you should only mock when necessary, and b) you should test the result, not the implementation. If the implementation changes (BigFunction uses something other than SmallFunction to do what it needs, E.g.: BigFunction adds the smallRow itself, in this example), that will break the test, despite the code working fine.

Answer 1

I think you are asking the wrong question - this should never be just a question of either - or.

You definitely need to test if BigFunction as a whole works as intended. You may do this test manually, either by yourself, or by handing it over to your testers, or by letting the users be the first ones to find out if BigFunction has a bug. But in case you don't want to do this manually, you have to write integration tests.

Be aware there are too many things which could be missed when you just mock out SmallFunction, solely relying on a unit test - for example:

• SmallFunction might be called, but it is called at the wrong place

• the "real" SmallFunction might have a necessary (or even worse: an unexpected) side effect which depends on the added row, and a mock would miss that

• by a change in the future, SmallFunction function throws intentionally an exception, which the unit test for SmallFunction takes care of - but since one forgot to make BigFunction deal with that exception, and since the mock for SmallFunction does not throw the exception, either, noone notices that the last line in BigFunctions will sometimes not be executed.

Hence, the real questions one should discuss here are:

• do I need an additional unit test for BigFunction, by mocking out SmallFunction?
• is the unit test for SmallFunction superfluous when there is already an automated integration test?

This was already discussed in-depth in the older SE question "Do I need unit test if I already have integration test?", and it is always a tradeoff (especially in this case, where some database operations may be part of BigFunction). You cannot answer this sensibly without knowing the whole context, the additional efforts to write the test, the benefits, the quality expectations for the system by different stakeholders etc.

About the argument of "duplicating the existing test": if the set-up and assertions for testing the success or failure of SmallFunction are simple, this should not be much of a problem. If the set-up or assertions are not so trivial, do in your tests what you always you do when find duplicate logic in your code: refactor the common logic to a central place. Then, if the logic of SmallFunction changes, there will be ideally only one place where you have to adapt the tests.

Answer 2

If we test the BigFunction, with the real SmallFunction, then we are doing an integration test.

So what are the pros and cons. Testing them together means that we can write less Unit tests (we could skip some of the unit tests of the smaller function). And yes, testing them together means we ensure that a working replacement of "SmallFunction" would not break the unit tests.

But let us zoom out a bit. There is the BIGGERFunction, which uses the BigFunction. If we apply the logic above, we could also run the tests without a mock of BigFunction. And then there is the EXTREMFunction which uses the BiggerFunction. And then there is the whole Application, which uses the EXTREMFunction... And then we are at System wide tests.

The disadvantage of mocking is, that you have to know the inner self of the Unit which we want to test. The advantage is, if the test breaks, we KNOW the problem is in that Unit. Therefore debugging gets much easier.

My personal opinion is, that mocking has multiple advantages. Finding the reason of a breaking test is easier. And in my experience i need more time for debugging, then for adapting unit tests because an implementation has changed. Also because mocking is work, i am motivated to invest a bit more thoughts into the code design, the application structure. Therefore i try to minimize the dependencies, i have to use more techniques like dependency injection to at least be able to mock. Long Story short, the decision to mock improves my productive code and it makes finding problems for me easier.

On the other side it is more work. That means if i am just fast wiring a prototype or only a very simple application where the structure is so easy that it does not need a lot of attention and the code itself is also quite simple (less bug prone), then i sometimes switch to less mocking. Because then at such a simple application the advantages i get with mocking are not that big, while i get a bit more speed.

But to tell the truth, most projects are (at least for me :-) ) not simple enough, so i stick with mocking most of the time.