How do you detect dependency problems with unit tests when you use mock objects?

Question

You have a class X and you write some unit tests that verify behaviour X1. There's also class A which takes X as a dependency.

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1. Time goes by, people do use your system, needs change, X evolves: you modify X to show behaviour X2. Obviously, unit tests for X will fail and you will need to adapt them.

But what with A? Unit tests for A will not fail when X's behaviour is modified (due to the mocking of X). How to detect that A's outcome will be different when run with the "real" (modified) X?

I'm expecting answers along the line of: "That's not the purpose of unit testing", but what value does unit testing have then? Does it really only tell you that when all tests pass, you haven't introduced a breaking change? And when some class's behaviour changes (willingly or unwillingly), how can you detect (preferably in an automated way) all the consequences? Shouldn't we focus more on integration testing?

Answer 1

When you write unit tests for A, you mock X

Do you? I don't, unless I absolutely have to. I have to if:

1. X is slow, or
2. X has side effects

If neither of these apply, then my unit tests of A will test X too. Doing anything else would be taking isolating tests to an illogical extreme.

If you have parts of your code using mocks of other parts of your code, then I'd agree: what is the point of such unit tests? So don't do this. Let those tests use the real dependencies as they form far more valuable tests that way.

And if some folk get upset with you calling these tests, "unit tests", then just call them "automated tests" and get on with writing good automated tests.

Answer 2

You need both. Unit tests to verify behavior of each of your units, and a few integration tests to make sure they're connected correctly. The problem with relying only on integration testing is combinatorial explosion resulting from interactions between all your units.

Let's say you have class A, that requires 10 unit tests to fully cover all paths. Then you have another class B, that also requires 10 unit tests to cover all the paths the code can take through it. Now let's say in your application, you need to feed the output of A into B. Now your code can take 100 different paths from the input of A to the output of B.

With unit tests, you only need 20 unit tests + 1 integration test to completely cover all the cases.

With integration tests, you will need 100 tests to cover all code paths.

Here's a very good video about the downsides of relying on integration tests only J B Rainsberger Integrated Tests Are A Scam HD

Answer 3

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1. Time goes by, people do use your system, needs change, X evolves: you modify X to show behaviour X2. Obviously, unit tests for X will fail and you will need to adapt them.

Woah, wait a moment. The implications of the tests for X failing are too important to gloss over like that.

If changing the implementation of X from X1 to X2 breaks the unit tests for X, that indicates that you've made a backwards incompatible change to the contract X.

X2 isn't an X, in the Liskov sense, so you should be thinking about other ways of meeting the needs of your stake holders (like introducing a new specification Y, that is implemented by X2).

For deeper insights, see Pieter Hinjens: The End of Software Versions, or Rich Hickey Simple Made Easy.

From the perspective of A, there is a precondition that the collaborator respects the contract X. And your observation is effectively that the isolated test for A doesn't give you any assurance that A recognizes collaborators that violate the X contract.

Review Integrated Tests are a Scam; in high level, you are expected to have as many isolated tests as you need to ensure that X2 implements the contract X correctly, and as many isolated tests as you need to ensure that A does the right thing given interesting responses from an X, and some smaller number of integrated tests to ensure that X2 and A agree on what X means.

You will sometimes see this distinction expressed as solitary tests vs sociable tests; see Jay Fields Working Effectively with Unit Tests.

Shouldn't we focus more on integration testing?

Again, see integrated tests are a scam - Rainsberger describes in detail a positive feedback loop that is common (in his experiences) to projects that are relying upon integrated (note spelling) tests. In summary, without the isolated/solitary tests applying pressure to the design, the quality degrades, leading to more mistakes and more integrated tests....

You will also need (some) integration tests. In addition to the complexity introduced by multiple modules, executing these tests tends to have more drag than the isolated tests; it's more efficient to be iterating on very fast checks when work is in progress, saving the additional checks for when you think you are "done".

Answer 4

Let me start by saying that the core premise of the question is flawed.

You are never testing (or mocking) implementations, you are testing (and mocking) interfaces.

If I have a real class X that implements the interface X1, I can write a mock XM that also complies with X1. Then my class A must use something that implements X1, which can either be class X or mock XM.

Now, suppose we change X to implement a new interface X2. Well, obviously my code no longer compiles. A requires something that implements X1, and that no longer exists. The issue has been identified and can be fixed.

Suppose instead of replacing X1, we just modify it. Now class A is all set. However, the mock XM no longer implements the interface X1. The issue has been identified and can be fixed.

---

The whole basis for unit testing and mocking is that you write code that uses interfaces. A consumer of an interface doesn't care how the code is implemented, only that the same contract is adhered to (inputs/outputs).

This breaks down when your methods have side effects, but I think that can safely be excluded as "cannot be unit tested or mocked".

Answer 5

Taking your questions in turn:

what value does unit testing have then

They're cheap to write and run and you get early feedback. If you break X, you'll find out more or less immediately if you have good tests. Don't even consider writing integration tests unless you've unit tested all your layers (yes, even on the database).

Does it really only tell you that when all tests pass, you haven't introduced a breaking change

Having tests that pass could actually tell you very little. You may not have written enough tests. You may not have tested enough scenarios. Code coverage can help here but it isn't a silver bullet. You may have tests that always pass. Hence red being the often overlooked first step of red, green, refactor.

And when some class's behaviour changes (willingly or unwillingly), how can you detect (preferably in an automated way) all the consequences

More testing - although tools are getting better and better. BUT you should be defining class behaviour in an interface (see below). N.B. there will always be a place for manual testing atop the testing pyramid.

Shouldn't we focus more on integration testing?

Ever more integration tests are not the answer either, they're expensive to write, run and maintain. Depending on your build setup, your build manager may exclude them anyway making them reliant on a developer remembering (never a good thing!).

I've seen developers spend hours trying to fix broken integration tests they'd have found in five minutes if they had good unit tests. Failing this, try just running the software - that is all your end users will care about. No point having million unit tests that pass if the whole house of cards falls down when the user runs the entire suite.

If you want to make sure class A consumes class X in the same way, you should be using an interface rather than a concretion. Then a breaking change is more likely to be picked up at compile time.

Answer 6

That's correct.

Unit tests are there to test the isolated functionality of a unit, a first-glance check that it works as intended and doesn't contain stupid errors.

Unit tests are not there to test that the entire application works.

What a lot of people forget is that unit tests are just the quickest and dirtiest means of validating your code. Once you know your little routines work, you then have to run Integration tests as well. Unit testing by itself is only marginally better than no testing.

The reason we have unit tests at all is that they're supposed to be cheap. Quick to create and run and maintain. Once you start turning them into min integration tests, you're into a world of pain. You might as well go full Integration test and ignore unit testing altogether if you're going to do that.

now, some people think that a unit isn't just a function in a class, but the whole class itself (myself included). However, all this does is increase the size of the unit so you might need less integration testing, but you still need it. It is still impossible to verify your program does what it is supposed to do without a full integration test suite.

and then, you'll still need to run the full integration testing on a live (or semi-live) system to check that it works with the conditions the customer uses.

Answer 7

Unit tests do not prove the correctness of anything. This is true for all tests. Usually unit tests are combined with contract-based design (design by contract is another way to say it) and possibly automated correctness proofs, if correctness needs to be verified on a regular basis.

If you have real contracts, consisting of class invariants, preconditions, and post conditions, it is possible to prove correctness hierarchically, by basing the correctness of higher level components on the contracts of lower level components. This the fundamental concept behind design by contract.

Answer 8

I find heavily mocked tests rarely useful. Most of the time, I end up reimplementing behaviour which the original class already has, which totally defeats the purpose of mocking.

M.e. a better strategy is to have good separation of concerns (e.g. you can test Part A of your app without bringing in parts B through Z). Such a good architecture really helps to write good test.

Also, I am more than willing to accept side effects as long as I can roll them back, e.g. if my method modifies data in the db, let it! As long as I can roll the db back to the previous state, what's the harm? Also, there is the benefit that my test can check if the data looks as expected. In-Memory DBs or specific test-versions of dbs really help here (e.g. RavenDBs in-memory test version).

Finally, I do like to do mocking on service boundaries, e.g. do not make that http call to service b, but let's intercept it and introduce an appropiate

Answer 9

I wish people in both camps would understand that class testing and behaviour testing are not orthogonal.

Class testing and unit testing are used interchangeably and they perhaps shouldn't be. Some unit tests just happen to be implemented in classes. That is all. Unit testing has happened for decades in languages without classes.

As for testing behaviours, it is perfectly possible to do this within class testing using the GWT construct.

Furthermore, whether your automated tests proceed along class or behaviour lines rather depends on your priorities. Some will need to rapidly prototype and get something out the door while others will have coverage constraints due to in house styles. Many reasons. They're both perfectly valid approaches. You pays your money, you takes your choice.

So, what to do when code breaks. If it has been coded to an interface, then just the concretion needs to change (along with any tests).

However, introducing a new behaviour needn't compromise the system at all. Linux et al are full of deprecated features. And things like constructors (and methods) can happily be overloaded without forcing all calling code to change.

Where class testing wins is where you need to make a change to a class that hasn't yet been plumbed in yet (due to time constraints, complexity or whatever). It is just so much easier to get started with a class if it has comprehensive tests.

Answer 10

Unless the interface for X changed, you do not need to change the unit test for A because nothing related to A has changed. It sounds like you really wrote a unit test of X and A together, but called it a unit test of A:

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1.

Ideally, the mock of X should simulate all possible behaviors of X, not just the behavior you expect from X. So no matter what you actually implement in X, A should already be capable of handling it. So no change to X, other than changing the interface itself, will have any effect on the unit test for A.

For example: Suppose A is a sorting algorithm, and A provides data to be sorted. The mock of X should provide a null return value, an empty list of data, a single element, multiple elements already sorted, multiple elements not already sorted, multiple elements sorted backwards, lists with the same element repeated, null values intermingled, a ridiculously large number of elements, and it should also throw an exception.

So perhaps X initially returned sorted data on Mondays and empty lists on Tuesdays. But now when X returns unsorted data on Mondays and throws exceptions on Tuesdays, A doesn't care -- those scenarios were already covered in A's unit test.

Answer 11

You have to look at different tests.

Unit tests themselves will only test X. They are there to prevent you changing the behavior of X but not secure the whole system. They make sure you can refactor you class without introducing a change in behavior. And if you break X, you broke it...

A should indeed mock X for its unit tests and the test with the Mock should keep passing even after you change it.

But there is more than one level of testing! There are also integration tests. These tests are there to validate the interaction between the classes. These tests usually have a higher price since they don't use mocks for everything. For example an integration test might actually write a record into a database, and a unit test should have no external dependencies.

Also if X needs to have a new behavior, it would be better to supply a new method that would supply the desired outcome