64

As I understand, the point of unit tests is to test units of code in isolation. This means, that:

  1. They should not break by any unrelated code change elsewhere in the codebase.
  2. Only one unit test should break by a bug in the tested unit, as opposed to integration tests (which may break in heaps).

All of this implies, that every outside dependency of a tested unit, should be mocked out. And I mean all the outside dependencies, not only the "outside layers" such as networking, filesystem, database, etc..

This leads to a logical conclusion, that virtually every unit test needs to mock. On the other hand, a quick Google search about mocking reveals tons of articles that claim that "mocking is a code smell" and should mostly (though not completely) be avoided.

Now, to the question(s).

  1. How should unit tests be written properly?
  2. Where exactly does the line between them and integration tests lie?

Update 1

Please consider the following pseudo code:

class Person {
    constructor(calculator) {}

    calculate(a, b) {
        const sum = this.calculator.add(a, b);

        // do some other stuff with the `sum`
    }
}

Can a test that tests the Person.calculate method without mocking the Calculator dependency (given, that the Calculator is a lightweight class that does not access "the outside world") be considered a unit test?

  • 9
    Part of this is just design experience that will come with time. You'll learn how to structure your components so that they do not have many difficult to mock dependencies. This means that testability must be a secondary design goal of any software. This goal is much easier to satisfy if test are written before or together with the code, e.g. using TDD and/or BDD. – amon Nov 27 '18 at 11:08
  • 31
    Put tests that run quickly and reliably into one folder. Put tests that are slow and potentially fragile into another. Run the tests in the first folder as often as possible (literally every time you pause in typing and the code compiles is the ideal, but not all dev environments support this). Run the slower tests less often (when you have a coffee break for example). Don't worry about unit and integration names. Call them fast and slow if you want. It doesn't matter. – David Arno Nov 27 '18 at 12:37
  • 4
    "that virtually every unit test needs to mock" Yeah, so? "a quick Google search about mocking reveals tons of articles that claim that "mocking is a code smell"" they're wrong. – Michael Nov 28 '18 at 13:34
  • 8
    @Michael Simply stating 'yeah, so' and declaring the opposing view wrong isn't a great way to approach a contentious subject like this. Perhaps write an answer and elaborate on why you think mocking should be everywhere, and perhaps why you think 'tons of articles' are inherently wrong? – AJFaraday Nov 28 '18 at 13:51
  • 4
    Since you didn't give a citation for "mocking is a code smell", I can only guess that you're misinterpreting what you read. Mocking is not a code smell. The need to use reflection or other shenanigans to inject your mocks is a code smell. The difficulty of mocking is inversely proportional to the quality of your API design. If you can write simple straightforward unit tests that just pass mocks to constructors, you're doing it right. – TKK Nov 28 '18 at 17:17
52

the point of unit tests is to test units of code in isolation.

Martin Fowler on Unit Test

Unit testing is often talked about in software development, and is a term that I've been familiar with during my whole time writing programs. Like most software development terminology, however, it's very ill-defined, and I see confusion can often occur when people think that it's more tightly defined than it actually is.

What Kent Beck wrote in Test Driven Development, By Example

I call them "unit tests", but they don't match the accepted definition of unit tests very well

Any given claim of "the point of unit tests is" will depend heavily on what definition of "unit test" is being considered.

If your perspective is that your program is composed of many small units that depend on one another, and if you constrain yourself to a style that tests each unit in isolation, then a lot of test doubles is an inevitable conclusion.

The conflicting advice that you see comes from people operating under a different set of assumptions.

For example, if you are writing tests to support developers during the process of refactoring, and splitting one unit into two is a refactoring that should be supported, then something needs to give. Maybe this kind of test needs a different name? Or maybe we need a different understanding of "unit".

You may want to compare:

Can a test that tests the Person.calculate method without mocking the Calculator dependency (given, that the Calculator is a lightweight class that does not access "the outside world") be considered a unit test?

I think that's the wrong question to ask; it's again an argument about labels, when I believe what we actually care about are properties.

When I'm introducing changes to the code, I don't care about isolation of tests -- I already know that "the mistake" is somewhere in my current stack of unverified edits. If I run the tests frequently, then I limit the depth of that stack, and finding the mistake is trivial (in the extreme case, the tests are run after every edit -- the max depth of the stack is one). But running the tests isn't the goal -- it's an interruption -- so there is value in reducing the impact of the interruption. One way of reducing the interruption is to ensure that the tests are fast (Gary Bernhardt suggests 300ms, but I haven't figured out how to do that in my circumstances).

If invoking Calculator::add doesn't significantly increase the time required to run the test (or any of the other important properties for this use case), then I wouldn't bother using a test double -- it doesn't provide benefits that outway the costs.

Notice the two assumptions here: a human being as part of the cost evaluation, and the short stack of unverified changes in the benefit evaluation. In circumstances where those conditions do not hold, the value of "isolation" changes quite a bit.

See also Hot Lava, by Harry Percival.

  • 4
    one thing that mocking add does do is prove that calculator can be mocked i.e. that the design does not couple person and calculator (though this can also be checked in other ways) – jk. Nov 28 '18 at 12:33
32

How exactly should unit tests be written without mocking extensively?

By minimising side-effects in your code.

Taking your example code, if calculator for example talks to a web API, then either you create fragile tests that rely on being able to interact with that web API, or you create a mock of it. If however its a deterministic, state-free set of calculation functions, then you don't (and shouldn't) mock it. If you do, you risk your mock behaving differently to the real code, leading to bugs in your tests.

Mocks should only be needed for code that read/writes to the file system, databases, URL endpoints etc; that are dependent on the environment you are running under; or that are highly stateful and non-deterministic in nature. So if you keep those parts of the code to a minimum and hide them behind abstractions, then they are easy to mock and the rest of your code avoids the need for mocks.

For the code points that do have side effects, it's worth writing tests that mock and tests that don't. The latter though need care as they will inherently be fragile and possibly slow. So you may want to only run them say overnight on a CI server, rather than every time you save and build your code. The former tests though should be run as often as practicable. As to whether each test is then a unit or integration test becomes academic and avoids "flame wars" over what is and isn't a unit test.

  • 7
    This is the correct answer both in practice and in terms of dodging a meaningless semantic debate. – Jared Smith Nov 27 '18 at 12:45
  • Do you have an example of a non-trivial open source codebase that uses such a style and still gets good test coverage? – Joeri Sebrechts Nov 27 '18 at 12:49
  • 3
    @JoeriSebrechts every single FP one? Example – Jared Smith Nov 27 '18 at 17:36
  • Not quite what I'm looking for, as that's just a collection of functions that are independent of each other, not a system of functions that call each other. How do you get around having to construct complex arguments to a function for the purpose of testing it, if that function is one of the top-level ones? E.g. the core loop of a game. – Joeri Sebrechts Nov 28 '18 at 8:16
  • @JoeriSebrechts hmm, either I'm misunderstanding what you want, or you didn't dig deep enough into the example I gave. The ramda functions use internal calls all over the place in their source (e.g. R.equals). Because these are for the most part pure functions, they generally aren't mocked out in the tests. – Jared Smith Nov 29 '18 at 13:40
30

These questions are quite different in their difficulty. Let's take question 2 first.

Unit tests and integration tests are clearly separated. A unit test tests one unit (method or class) and uses other units only as much as necessary to achieve that goal. Mocking may be necessary, but it is not the point of the test. An integration test tests the interaction between different actual units. This difference is the entire reason why we need both unit and integration testing - if one did the job of the other well enough, we wouldn't, but it's turned out that it's usually more efficient to use two specialized tools rather than one generalized tool.

Now for the important question: How should you unit test? As said above, unit tests should construct auxiliary structures only as far as necessary. Often it is easier to use a mock database than your real database or even any real database. However, mocking in itself has no value. If often happens that it is in fact easier to use actual components of another layer as input for a mid-level unit test. If so, don't hesitate to use them.

Many practitioners are afraid that if unit test B reuses classes that were already tested by unit test A, then a defect in unit A causes test failures in multiple places. I consider this not a problem: a test suite has to succeed 100% in order to give you the reassurance you need, so it is not a big problem to have too many failures - after all, you do have a defect. The only critical problem would be if a defect triggered too few failures.

Therefore, don't make a religion of mocking. It is a means, not an end, so if you can get away with avoiding the extra effort, you should do so.

  • 3
    The only critical problem would be if a defect triggered too few failures. this is one of the weak points of mocking. We have to "programme" the expected behaviour so, we might fail at doing so, causing our tests to end as "false positives". But mocking is a very useful technique in order to achieve determinism (the most important condition of testing). I use them in all my projects when possible. They also show me when the integration is too complex or the dependency too tight. – Laiv Nov 27 '18 at 11:08
  • 1
    If a unit that's being tested uses other units, isn't it really an integration test? Because in essence this unit would be testing the interaction between the said units, exactly like an integration test would. – Alexander Lomia Nov 27 '18 at 11:11
  • 9
    @AlexanderLomia: What would you call a unit? Would you call 'String' a unit as well? I would, but I wouldn't dream of mocking it. – Bart van Ingen Schenau Nov 27 '18 at 11:36
  • 3
    "Unit tests and integration tests are clearly separated. A unit test tests one unit (method or class) and uses other units only as much as necessary to achieve that goal". Here's the rub. That's your definition of a unit test. Mine is quite different. So the distinction between them is only "clearly separated" for any given definition but the separation varies between definitions. – David Arno Nov 27 '18 at 12:20
  • 3
    @Voo Having worked with such codebases, whilst it can be a nuisance to find the original problem (especially if the architecture has painted over the things you'd use to debug it), I've still had more trouble from mocks that caused the tests to keep working after the code they were used to test had broken. – James_pic Nov 27 '18 at 17:05
4

OK, so to answer your questions directly:

How should unit tests be written properly?

As you say, you should be mocking dependencies and testing just the unit in question.

Where exactly does the line between them and integration tests lie?

An Integration test is a unit test where your dependencies are not mocked.

Can a test that tests the Person.calculate method without mocking the Calculator be considered a unit test?

No. You need to inject the calculator dependency into this code and you have a choice between a mocked version or a real one. If you use a mocked one its a unit test, if you use a real one its an integration test.

However, a caveat. do you really care what people think your tests should be called?

But your real question seems to be this:

a quick Google search about mocking reveals tons of articles that claim that "mocking is a code smell" and should mostly (though not completely) be avoided.

I think the problem here is that a lot of people use mocks to completely recreate the dependencies. For example I might mock the calculator in your example as

public class MockCalc : ICalculator
{
     public Add(int a, int b) { return 4; }
}

I would not do something like:

myMock = Mock<ICalculator>().Add((a,b) => {return a + b;})
myPerson.Calculate()
Assert.WasCalled(myMock.Add());

I would argue that, that would be "testing my mock" or "testing the implementation". I would say "Don't write Mocks! *like that".

Other people would disagree with me, we would start massive flame wars on our blogs about the Best way to Mock, which really would make no sense unless you understood the whole background of the various approaches and really don't offer a whole lot of value to someone who just wants to write good tests.

  • Thanks for an exhaustive answer. As of caring about what other people think about my tests - actually I want to avoid writing semi-integration, semi-unit tests that tend to become an unreliably mess as the project progresses. – Alexander Lomia Nov 27 '18 at 12:23
  • no probs, I think the problem is that the definitions of the two things aren't 100% agreed on by everyone. – Ewan Nov 27 '18 at 12:32
3
  1. How should unit tests be implemented properly?

My rule of thumb is that proper unit tests:

  • Are coded against interfaces, not implementations. This has many benefits. For one, it ensures that your classes follow the Dependency Inversion Principle from SOLID. Also, this is what your other classes do (right?) so your tests should do the same. Also, this allows you to test multiple implementations of the same interface while reusing much of the test code (only initialization and some assertions would change).
  • Are self-contained. As you said, changes in any outside code cannot affect the test result. As such, unit tests can execute at build-time. This means you need mocks to remove any side effects. However, if you are following the Dependency Inversion Principle, this should be relatively easy. Good test frameworks like Spock can be used to dynamically provide mock implementations of any interface to use in your tests with minimal coding. This means that each test class only needs to exercise code from exactly one implementation class, plus the test framework (and maybe model classes ["beans"]).
  • Do not require a separate running application. If the test needs to "talk to something", whether a database or a web service, it's an integration test, not a unit test. I draw the line at network connections or the filesystem. A purely in-memory SQLite database, for example, is fair game in my opinion for a unit test if you really need it.

If there are utility classes from frameworks that complicate unit testing, you may even find it useful to create very simple "wrapper" interfaces and classes to facilitate mocking of those dependencies. Those wrappers would then not necessarily be subject to unit tests.

  1. Where exactly does the line between them [unit tests] and integration tests lie?

I have found this distinction to be the most useful:

  • Unit tests simulate "user code", verifying behavior of implementation classes against the desired behavior and semantics of code-level interfaces.
  • Integration tests simulate the user, verifying behavior of the running application against specified use cases and/or formal APIs. For a web service, the "user" would be a client application.

There is gray area here. For example, if you can run an application in a Docker container and run the integration tests as the final stage of a build, and destroy the container afterwards, is it OK to include those tests as "unit tests"? If this is your burning debate, you're in a pretty good place.

  1. Is it true that virtually every unit test needs to mock?

No. Some individual test cases will be for error conditions, like passing null as a parameter and verifying you get an exception. Lots of tests like that will not require any mocks. Also, implementations that have no side effects, for example string processing or math functions, may not require any mocks because you simply verify the output. But most classes worth having, I think, will require at least one mock somewhere in the test code. (The fewer, the better.)

The "code smell" issue you mentioned arises when you have a class that is overly complicated, that requires a long list of mock dependencies in order to write your tests. This is a clue that you need to refactor the implementation and split things up, so that each class has a smaller footprint and a clearer responsibility, and is therefore more easily testable. This will improve quality in the long run.

Only one unit test should break by a bug in the tested unit.

I don't think this is a reasonable expectation, because it works against reuse. You may have a private method, for example, that is called by multiple public methods published by your interface. A bug introduced into that one method might then cause multiple test failures. This doesn't mean you should copy the same code into each public method.

3
  1. They should not break by any unrelated code change elsewhere in the codebase.

I'm not really sure how this rule is useful. If a change in one class/method/whatever can break the behaviour of another in production code, then the things are, in reality, collaborators, and not unrelated. If your tests break and your production code doesn't, then your tests are suspect.

  1. Only one unit test should break by a bug in the tested unit, as opposed to integration tests (which may break in heaps).

I'd regard this rule with suspicion too. If you're really good enough to structure your code and write your tests such that one bug causes exactly one unit test failure, Then you're saying you've identified all the potential bugs already, even as the codebase evolves to use cases you haven't anticipated.

Where exactly does the line between them and integration tests lie?

I don't think that's an important distinction. What is a 'unit' of code anyhow?

Try to find entry points at which you can write tests that just 'make sense' in terms of the problem domain/business rules that that level of the code is dealing with. Often these tests are somewhat 'functional' in nature - put in an input, and test that the output is as expected. If the tests express a desired behaviour of the system, then they often remain quite stable even as the production code evolves and is refactored.

How exactly should unit tests be written without mocking extensively?

Don't read too much into the word 'unit', and lean towards using your real production classes in tests, without worrying too much if you're involving more than one of them in a test. If one of them is hard to use (because it takes a lot of initialisation, or it needs to hit a real database/email server etc), then let your thoughts turn to mocking/faking.

  • "What is a 'unit' of code anyhow?" very good question that can have unexpected answers that might even depend on who is answering. Typically, most definitions of unit tests explain them as relating to a method or a class but that's not a really useful measure of a "unit" in all cases. If I have a Person:tellStory() method incorporates a person's details into a string then returns that, then the "story" is probably one unit. If I make a private helper method that tucks away some of the code, then I don't believe I've introduced a new unit - I don't need to test that separately. – vlaz Nov 29 '18 at 6:57

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.