96

You have a class X and you write some unit tests that verify behaviour X1. There's also class A which takes X as a dependency.

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1. Time goes by, people do use your system, needs change, X evolves: you modify X to show behaviour X2. Obviously, unit tests for X will fail and you will need to adapt them.

But what with A? Unit tests for A will not fail when X's behaviour is modified (due to the mocking of X). How to detect that A's outcome will be different when run with the "real" (modified) X?

I'm expecting answers along the line of: "That's not the purpose of unit testing", but what value does unit testing have then? Does it really only tell you that when all tests pass, you haven't introduced a breaking change? And when some class's behaviour changes (willingly or unwillingly), how can you detect (preferably in an automated way) all the consequences? Shouldn't we focus more on integration testing?

  • 9
    Possible duplicate of (Why) is it important that a unit test not test dependencies? – gnat Mar 27 '18 at 11:54
  • 36
    In addition to all the answers suggested, I must say I take issue with the following statement: 'Does it really only tell you that when all tests pass, you haven't introduced a breaking change?' If you really think removing the fear of refactoring is of little value, you are on the fast track towards writing unmaintainable code – crizzis Mar 27 '18 at 15:13
  • 5
    Unit testing tells you whether your unit of code behaves as you expect it to. No more or less. Mocks and test doubles provide an artificial, controlled environment for you to exercise your unit of code (in isolation) to see if it meets your expectations. No more or less. – Robert Harvey Mar 27 '18 at 17:37
  • 2
    I believe your premise is incorrect. When you mention X1 you are saying that X implements interface X1. If you change the interface X1 to X2 the mock you used in the other tests should not compile anymore, hence you are forced to fix those tests too. Changes in the class behaviour should not matter. In fact, your class A should not depend on implementation details (which is what you'd be changing in that case). So the unit tests for A are still correct, and they tell you that A works given an ideal implementation of the interface. – Bakuriu Mar 27 '18 at 17:49
  • 5
    I don't know about you, but when I have to work on a codebase with no tests, I'm scared to death I'm going to break something. And why? Because it happens so often that something breaks when it wasn't intended for. And bless our tester's hearts, they can't test everything. Or even close. But a unit test will happily chug away through boring routine after boring routine. – corsiKa Mar 27 '18 at 21:26

11 Answers 11

125

When you write unit tests for A, you mock X

Do you? I don't, unless I absolutely have to. I have to if:

  1. X is slow, or
  2. X has side effects

If neither of these apply, then my unit tests of A will test X too. Doing anything else would be taking isolating tests to an illogical extreme.

If you have parts of your code using mocks of other parts of your code, then I'd agree: what is the point of such unit tests? So don't do this. Let those tests use the real dependencies as they form far more valuable tests that way.

And if some folk get upset with you calling these tests, "unit tests", then just call them "automated tests" and get on with writing good automated tests.

  • 94
    @Laiv, No, unit tests are supposed to act as a unit, ie run in isolation, from other tests. Nodes and graphs can take a hike. If I can run an isolated, side-effect free end-to-end test in a short time, that's a unit test. In if you don't like that definition, call it an automated test and stop writing crap tests to fit silly semantics. – David Arno Mar 27 '18 at 12:36
  • 9
    @DavidArno There is alas a very broad definition of isolated. Some would like a "unit" to include the mid-tier and the database. They can believe whatever they like but the chances are that on a development of any size, the wheels will come off in fairly short order as the build manager will throw it out. Generally, if they're isolated to the assembly (or equivalent), that is fine. N.B. if you code to interfaces then it is far easier to add mocking and DI later. – Robbie Dee Mar 27 '18 at 12:47
  • 13
    You're advocating for a different kind of test rather than answering the question. Which is a valid point, but this is a pretty underhanded way of doing it. – Phil Frost Mar 27 '18 at 23:53
  • 17
    @PhilFrost, to quote myself, "And if some folk get upset with you calling these tests, "unit tests", then just call them "automated tests" and get on with writing good automated tests." Write useful tests, not silly tests that simply meet some random definition of a word. Or alternatively, accept that maybe you have your definition of "unit test" wrong and that you are over using mocks because you have it wrong. Either way, you'll end up with better tests. – David Arno Mar 28 '18 at 7:21
  • 30
    I'm with @DavidArno on this one; my testing strategy changed after watching this talk by Ian Cooper: vimeo.com/68375232. To boil it down to a single sentence: Don't test classes. Test behaviours. Your tests should not have knowledge of the internal classes / methods used to implement the desired behaviour; they should only know the public surface of your API / library and they should test that. If tests have too much knowledge then you're testing implementation details and your tests become brittle, coupled to your implementation, and actually just an anchor around your neck. – Richiban Mar 28 '18 at 12:33
77

You need both. Unit tests to verify behavior of each of your units, and a few integration tests to make sure they're connected correctly. The problem with relying only on integration testing is combinatorial explosion resulting from interactions between all your units.

Let's say you have class A, that requires 10 unit tests to fully cover all paths. Then you have another class B, that also requires 10 unit tests to cover all the paths the code can take through it. Now let's say in your application, you need to feed the output of A into B. Now your code can take 100 different paths from the input of A to the output of B.

With unit tests, you only need 20 unit tests + 1 integration test to completely cover all the cases.

With integration tests, you will need 100 tests to cover all code paths.

Here's a very good video about the downsides of relying on integration tests only J B Rainsberger Integrated Tests Are A Scam HD

  • 1
    I'm sure it is no coincidence that the question mark over the efficacy of integration tests has gone hand in hand with unit tests covering every more layers. – Robbie Dee Mar 27 '18 at 13:00
  • 16
    Right, but nowhere does your 20 unit test's require mocking. If you have your 10 tests of A that cover all of A and your 10 tests that cover all of B and also retest 25% of A as a bonus this seems between "fine" and a good thing. Mocking A in Bs tests seems actively stupid (unless there really is a reason why A is a problem, e.g. it's the database or it brings in a long web of other things) – Richard Tingle Mar 27 '18 at 21:22
  • 9
    I disagree with the idea that a single integration test is sufficient if you want full coverage. The reactions of B to what A outputs will vary based on the output; if changing a parameter in A changes its output, then B may not handle it correctly. – Matthieu M. Mar 28 '18 at 9:09
  • 3
    @Eternal21: My point was that sometimes, the problem is not in the individual behavior, but in an unexpected interaction. That is, when the glue between A and B behaves unexpectedly in some scenario. So both A and B act according to specifications, and the happy case works, but on some inputs there's a bug in the glue code... – Matthieu M. Mar 28 '18 at 11:53
  • 1
    @MatthieuM. I’d argue this is beyond the scope of Unit Testing. The glue code can be unit tested itself while the interactions between A and B via the glue code is an Integration Test. When specific edge cases or bugs are found they can be added to the glue code unit tests and ultimately verified in Integration testing. – Andrew T Finnell Apr 1 '18 at 20:25
72

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1. Time goes by, people do use your system, needs change, X evolves: you modify X to show behaviour X2. Obviously, unit tests for X will fail and you will need to adapt them.

Woah, wait a moment. The implications of the tests for X failing are too important to gloss over like that.

If changing the implementation of X from X1 to X2 breaks the unit tests for X, that indicates that you've made a backwards incompatible change to the contract X.

X2 isn't an X, in the Liskov sense, so you should be thinking about other ways of meeting the needs of your stake holders (like introducing a new specification Y, that is implemented by X2).

For deeper insights, see Pieter Hinjens: The End of Software Versions, or Rich Hickey Simple Made Easy.

From the perspective of A, there is a precondition that the collaborator respects the contract X. And your observation is effectively that the isolated test for A doesn't give you any assurance that A recognizes collaborators that violate the X contract.

Review Integrated Tests are a Scam; in high level, you are expected to have as many isolated tests as you need to ensure that X2 implements the contract X correctly, and as many isolated tests as you need to ensure that A does the right thing given interesting responses from an X, and some smaller number of integrated tests to ensure that X2 and A agree on what X means.

You will sometimes see this distinction expressed as solitary tests vs sociable tests; see Jay Fields Working Effectively with Unit Tests.

Shouldn't we focus more on integration testing?

Again, see integrated tests are a scam - Rainsberger describes in detail a positive feedback loop that is common (in his experiences) to projects that are relying upon integrated (note spelling) tests. In summary, without the isolated/solitary tests applying pressure to the design, the quality degrades, leading to more mistakes and more integrated tests....

You will also need (some) integration tests. In addition to the complexity introduced by multiple modules, executing these tests tends to have more drag than the isolated tests; it's more efficient to be iterating on very fast checks when work is in progress, saving the additional checks for when you think you are "done".

  • 8
    This should be the accepted answer. The question outlines a situation where the behaviour of a class has been modified in an incompatible way, but still outwardly looks identical. The problem here lies in the design of the application, not the unit tests. The way this circumstance would be caught in testing is using an integration test between those two classes. – Nick Coad Mar 29 '18 at 3:30
  • 1
    "without the isolated/solitary tests applying pressure to the design, the quality degrades". I think this is an important point. Alongside with behaviors checks, unit-tests have the side effect to force you to have a more modular design. – MickaëlG Mar 30 '18 at 12:07
15

Let me start by saying that the core premise of the question is flawed.

You are never testing (or mocking) implementations, you are testing (and mocking) interfaces.

If I have a real class X that implements the interface X1, I can write a mock XM that also complies with X1. Then my class A must use something that implements X1, which can either be class X or mock XM.

Now, suppose we change X to implement a new interface X2. Well, obviously my code no longer compiles. A requires something that implements X1, and that no longer exists. The issue has been identified and can be fixed.

Suppose instead of replacing X1, we just modify it. Now class A is all set. However, the mock XM no longer implements the interface X1. The issue has been identified and can be fixed.


The whole basis for unit testing and mocking is that you write code that uses interfaces. A consumer of an interface doesn't care how the code is implemented, only that the same contract is adhered to (inputs/outputs).

This breaks down when your methods have side effects, but I think that can safely be excluded as "cannot be unit tested or mocked".

  • 9
    This answer makes lots of assumptions that need not hold. Firstly, it assumes, roughly, that we are in C# or Java (or, more precisely, that we're in a compiled language, that the language has interfaces, and that X implements an interface; none of these need be true). Secondly, it assumes that any change to the behaviour or "contract" of X requires a change to the interface (as understood by the compiler) that X implements. This is plainly not true, even if we are in Java or C#; you can change a method implementation without changing its signature. – Mark Amery Mar 28 '18 at 12:33
  • 6
    @MarkAmery It's true that the "interface" terminology is more specific to C# or Java, but I think the point stands assuming a defined "contract" of behavior (and if that isn't codified then automatically detecting this is impossible). You're also completely correct that an implementation can be changed without a change to the contract. But a change to implementation without a change in interface (or contract) shouldn't impact any consumer. If the behavior of A depends on how the interface (or contract) is implemented, then it's impossible to (meaningfully) unit test. – Vlad274 Mar 28 '18 at 12:47
  • 1
    "You're also completely correct that an implementation can be changed without a change to the contract" - while also true, this isn't the point I was trying to make. Rather, I'm asserting a distinction between the contract (a programmer's understanding of what an object is supposed to do, perhaps specified in documentation) and the interface (a list of method signatures, understood by the compiler) and saying that the contract can be changed without changing the interface. All functions with the same signature are interchangeable from the type system's perspective, but not in reality! – Mark Amery Mar 28 '18 at 12:55
  • 4
    @MarkAmery: I don't think Vlad is using the word "interface" in the same sense as you're using it; the way I read the answer, it's not talking about interfaces in the narrow C#/Java sense (i.e. a set of method signatures) but in the general sense of the word, as used e.g. in the terms "application programming interface" or even "user interface". [...] – Ilmari Karonen Mar 29 '18 at 14:33
  • 6
    @IlmariKaronen If Vlad is using "interface" to mean "contract" rather than in the narrow C#/Java sense, then the statement "Now, suppose we change X to implement a new interface X2. Well, obviously my code no longer compiles." is just plain false, since you can change a contract without changing any method signatures. But honestly, I think the problem here is that Vlad isn't using either meaning consistently, but is conflating them - which is what leads down the path of claiming that any change to contract X1 will necessarily cause a compilation error without noticing that that's false. – Mark Amery Mar 29 '18 at 14:42
9

Taking your questions in turn:

what value does unit testing have then

They're cheap to write and run and you get early feedback. If you break X, you'll find out more or less immediately if you have good tests. Don't even consider writing integration tests unless you've unit tested all your layers (yes, even on the database).

Does it really only tell you that when all tests pass, you haven't introduced a breaking change

Having tests that pass could actually tell you very little. You may not have written enough tests. You may not have tested enough scenarios. Code coverage can help here but it isn't a silver bullet. You may have tests that always pass. Hence red being the often overlooked first step of red, green, refactor.

And when some class's behaviour changes (willingly or unwillingly), how can you detect (preferably in an automated way) all the consequences

More testing - although tools are getting better and better. BUT you should be defining class behaviour in an interface (see below). N.B. there will always be a place for manual testing atop the testing pyramid.

Shouldn't we focus more on integration testing?

Ever more integration tests are not the answer either, they're expensive to write, run and maintain. Depending on your build setup, your build manager may exclude them anyway making them reliant on a developer remembering (never a good thing!).

I've seen developers spend hours trying to fix broken integration tests they'd have found in five minutes if they had good unit tests. Failing this, try just running the software - that is all your end users will care about. No point having million unit tests that pass if the whole house of cards falls down when the user runs the entire suite.

If you want to make sure class A consumes class X in the same way, you should be using an interface rather than a concretion. Then a breaking change is more likely to be picked up at compile time.

9

That's correct.

Unit tests are there to test the isolated functionality of a unit, a first-glance check that it works as intended and doesn't contain stupid errors.

Unit tests are not there to test that the entire application works.

What a lot of people forget is that unit tests are just the quickest and dirtiest means of validating your code. Once you know your little routines work, you then have to run Integration tests as well. Unit testing by itself is only marginally better than no testing.

The reason we have unit tests at all is that they're supposed to be cheap. Quick to create and run and maintain. Once you start turning them into min integration tests, you're into a world of pain. You might as well go full Integration test and ignore unit testing altogether if you're going to do that.

now, some people think that a unit isn't just a function in a class, but the whole class itself (myself included). However, all this does is increase the size of the unit so you might need less integration testing, but you still need it. It is still impossible to verify your program does what it is supposed to do without a full integration test suite.

and then, you'll still need to run the full integration testing on a live (or semi-live) system to check that it works with the conditions the customer uses.

2

Unit tests do not prove the correctness of anything. This is true for all tests. Usually unit tests are combined with contract-based design (design by contract is another way to say it) and possibly automated correctness proofs, if correctness needs to be verified on a regular basis.

If you have real contracts, consisting of class invariants, preconditions, and post conditions, it is possible to prove correctness hierarchically, by basing the correctness of higher level components on the contracts of lower level components. This the fundamental concept behind design by contract.

  • Unit tests do not prove the correctness of anything. Not sure I understand this, unit tests check their own results surely? Or did you mean a behaviour cannot be proved correct since this may encompass multiple layers? – Robbie Dee Mar 28 '18 at 17:56
  • 7
    @RobbieDee I guess, he meant that when you test for fac(5) == 120, you have not proven that fac() does indeed return the factorial of its argument. You have only proven that fac() returns the factorial of five when you pass in 5. And even that is not certain, as fac() could conceivably return 42 instead on the first mondays after a total eclipse in Timbuktu... The problem here is, that you cannot prove compliance by checking individual test inputs, you would need to check all possible inputs, and also prove that you have not forgotten any (like reading the system clock). – cmaster Mar 29 '18 at 10:08
  • 1
    @RobbieDee Tests (including unit tests) are a poor substitute, often the best available, for the real goal, a machine checked proof. Consider the entire state space of the units under test, including the state space of any components or mocks therein. Unless you have a very restricted state space, tests cannot cover that state space. Complete coverage would be a proof, but this is only available for tiny state spaces, for example, testing a single object containing a single mutable byte or 16 bit integer. Automated proofs are vastly more valuable. – Frank Hileman Mar 29 '18 at 15:56
  • 1
    @cmaster You summarized the difference between a test and a proof very well. Thanks! – Frank Hileman Mar 29 '18 at 15:57
2

I find heavily mocked tests rarely useful. Most of the time, I end up reimplementing behaviour which the original class already has, which totally defeats the purpose of mocking.

M.e. a better strategy is to have good separation of concerns (e.g. you can test Part A of your app without bringing in parts B through Z). Such a good architecture really helps to write good test.

Also, I am more than willing to accept side effects as long as I can roll them back, e.g. if my method modifies data in the db, let it! As long as I can roll the db back to the previous state, what's the harm? Also, there is the benefit that my test can check if the data looks as expected. In-Memory DBs or specific test-versions of dbs really help here (e.g. RavenDBs in-memory test version).

Finally, I do like to do mocking on service boundaries, e.g. do not make that http call to service b, but let's intercept it and introduce an appropiate

1

I wish people in both camps would understand that class testing and behaviour testing are not orthogonal.

Class testing and unit testing are used interchangeably and they perhaps shouldn't be. Some unit tests just happen to be implemented in classes. That is all. Unit testing has happened for decades in languages without classes.

As for testing behaviours, it is perfectly possible to do this within class testing using the GWT construct.

Furthermore, whether your automated tests proceed along class or behaviour lines rather depends on your priorities. Some will need to rapidly prototype and get something out the door while others will have coverage constraints due to in house styles. Many reasons. They're both perfectly valid approaches. You pays your money, you takes your choice.

So, what to do when code breaks. If it has been coded to an interface, then just the concretion needs to change (along with any tests).

However, introducing a new behaviour needn't compromise the system at all. Linux et al are full of deprecated features. And things like constructors (and methods) can happily be overloaded without forcing all calling code to change.

Where class testing wins is where you need to make a change to a class that hasn't yet been plumbed in yet (due to time constraints, complexity or whatever). It is just so much easier to get started with a class if it has comprehensive tests.

  • Where you have concretion I would have written implementation. Is this a calque from another language (I would guess French) or is there an important distinction between concretion and implementation? – Peter Taylor Mar 29 '18 at 11:44
0

Unless the interface for X changed, you do not need to change the unit test for A because nothing related to A has changed. It sounds like you really wrote a unit test of X and A together, but called it a unit test of A:

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1.

Ideally, the mock of X should simulate all possible behaviors of X, not just the behavior you expect from X. So no matter what you actually implement in X, A should already be capable of handling it. So no change to X, other than changing the interface itself, will have any effect on the unit test for A.

For example: Suppose A is a sorting algorithm, and A provides data to be sorted. The mock of X should provide a null return value, an empty list of data, a single element, multiple elements already sorted, multiple elements not already sorted, multiple elements sorted backwards, lists with the same element repeated, null values intermingled, a ridiculously large number of elements, and it should also throw an exception.

So perhaps X initially returned sorted data on Mondays and empty lists on Tuesdays. But now when X returns unsorted data on Mondays and throws exceptions on Tuesdays, A doesn't care -- those scenarios were already covered in A's unit test.

-2

You have to look at different tests.

Unit tests themselves will only test X. They are there to prevent you changing the behavior of X but not secure the whole system. They make sure you can refactor you class without introducing a change in behavior. And if you break X, you broke it...

A should indeed mock X for its unit tests and the test with the Mock should keep passing even after you change it.

But there is more than one level of testing! There are also integration tests. These tests are there to validate the interaction between the classes. These tests usually have a higher price since they don't use mocks for everything. For example an integration test might actually write a record into a database, and a unit test should have no external dependencies.

Also if X needs to have a new behavior, it would be better to supply a new method that would supply the desired outcome

protected by gnat Mar 29 '18 at 8:23

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.