3

tl;dr: Is it a unit test or an integration test, to ensure that the inputs to a mocked dependency were correct?

More details:

Suppose I'm given a requirement like this:

Create a function that returns the current temperature somewhere on Earth, given the latitude and longitude of that place. Signature has to be int getTemp(double latitude, double longitude).

Of course I have to consult an external service to get that value. One that's available (SuperWeather) has an appropriate API I can call on, and its signature is WeatherInfo getWeather(double longitude, double latitude) (note the reversed order of lat/lon).

I set up a mock of SuperWeather for my unit tests, and run a test like this:

void ReturnsTemperature()
{
    getWeatherMock.returns(new WeatherInfo() { temperature = 17 });

    int result = SUT(1.2, 3.4);

    assert(result).equals(17);
}

The thing is that this test does not make sure I pass in the correct location to SuperWeather. I could write my function give the latitude first while it's supposed to be second (or even make it pass hard-coded values instead of its inputs) and the test would pass.

So I'd argue that I should add another test, like

void GetsWeatherInfo()
{
    SUT(1.2, 3.4);

    assert(getWeatherMock).wasCalledWith(3.4, 1.2);
}

Discussing the above with a colleague, I was told that the latter is an integration test, and if I've made a mistake with the parameters to SuperWeather I should let the person doing integration tests find this out.

I argued that as long as there is a reasonable chance that the unit I wrote has a problem in it and said problem does not depend on what other units do, there should be a test that will catch that problem. The second test is still testing the unit I wrote, not anything else.

Does using a dependency make it an integration test, even if the behaviour of the dependency is not being tested?

2
  • 1
    You're not using a dependency, i don't think you can reasonably describe that as an integration test. You're testing the system against a test double. Either you need to test it was called with the right arguments, or set up a test double that doesn't respond (or throws an error) if it was called with anything unexpected. Otherwise you're only testing half of what the SUT does. – jonrsharpe Apr 15 at 22:01
  • "I was told that the latter is an integration test" - that is plain wrong. "if I've made a mistake with the parameters to SuperWeather I should let the person doing integration tests find this out" - maybe, but not the reason given. – Doc Brown Apr 16 at 5:35
6

Is it a unit test or an integration test, to ensure that the inputs to a mocked dependency were correct?

If "unit test" and "integration test" are the only two choices you are allowed, then you should probably consider this to be a unit test, because your test includes only one of the two elements that participate in the integration.

If you look at the Sandi Metz talk Magic Tricks of Testing, she makes an argument that you shouldn't be unit testing this interaction at all. The argument is that you are querying WeatherInfo, so all you really need for unit testing is a stub that returns interesting canned answers.

That said, it's not entirely unreasonable to check that the right arguments are being passed in the right order, and a mock or a spy is the usual way to achieve that. We're still broadly within the umbrella of unit tests here.

That said, you should be aware of the fact that your check doesn't show that the arguments are in the right order, but only that your future refactorings leave the arguments in the same order. That's not nothing, but it doesn't really protect you against either (a) your having the arguments in the wrong order or (b) the order of arguments being changed at the other end.

if I've made a mistake with the parameters to SuperWeather I should let the person doing integration tests find this out.

I'd counter that argument with a discussion of the investment odds. Yes, you could wait for feedback from some other tester. But the test you propose is cheap, and gives you feedback about refactoring mistakes much more quickly, at relatively small cost. Unless the person doing the integration tests is much cheaper or much faster, you've got maths on your side.

A more interesting question is whether you should instead be changing the design so that you don't need to be leaning on testing quite so hard. For instance, if your code is type checked, then you could use different types for Latitude and Longitude, and the risks of introducing an undetected error drop dramatically.


"That said, you should be aware of the fact that your check doesn't show that the arguments are in the right order, but only that your future refactorings leave the arguments in the same order." Could you explain this a bit better?

When we write a test using a substitute for a dependency (which might be a test double, or might be an alternative "production" implementation), what we end up with is a check that the implementation is consistent with the substitute.

Assertion:
a === c

Proof:
a === b
b === c

The check that we want is that our code works with the remote api a === c. The check that we have is that our code works with the substitute a === b.

So we need something else, something that plays the b === c role, to decide if that's "good enough".

Now, consider a model for a test written in the past. We're using today's implementation of our unit, but yesterday's substitute, as a proxy for today's 3rd party api...

Assertion:
today(a) === today(c)

Proof:
today(a) === yesterday(b)
yesterday(b) === today(c)

Passing yesterday's test gives us strong evidence of today(a) === yesterday(b) - measurements of our code taken today match those taken yesterday. But we still need something to bridge the "is that good enough" question if we want to claim today(a) === today(c).

In most cases, that second claim is something like yesterday(b) === yesterday(c) AND c is supposed to be stable. We don't have to drop a ball everyday to check that gravity still works.

5
  • Honestly, an API using different types for Latitude and Longitude would immediately trigger my "WTF" counter when I had to review such code. I guess from a practical perspectice this would violate the expections of almost every one who uses this API (or the "principle of least astonishment", if you prefer a fancy term for this). +1 either. – Doc Brown Apr 16 at 5:28
  • "That said, you should be aware of the fact that your check doesn't show that the arguments are in the right order, but only that your future refactorings leave the arguments in the same order." Could you explain this a bit better? In the scenario, I've written the test based on the API's documentation. – George T Apr 16 at 6:14
  • 3
    Instead of two separate types maybe a Point/Location/Coordinate type having members Latitude and Longitude. – r_ahlskog Apr 16 at 7:02
  • @GeorgeT but what if the docs are wrong? Or you hadn't checked them, and made an incorrect assumption about that API? Or it's correct now but changes later on (particularly a problem with third party dependencies, which is why you shouldn't mock what you don't own). If you're only testing with test doubles, you're very reliant on them being correct; otherwise your code could pass the tests but not actually work. Here's an example I spotted where the proposed test double was wrong: stackoverflow.com/a/65627662/3001761. – jonrsharpe Apr 16 at 8:18
  • 1
    @GeorgeT I added more explanation; let me know if it is better explanation. – VoiceOfUnreason Apr 16 at 12:57
2

Testing interaction with dependencies

Should unit tests assert the inputs to dependencies?

Answering the question in the title, it's a clear yes.

To make that visible, let's use the example of a NameReportingService which takes in the name of a person, and is expected to write two things to the database: the person's initial, and the amount of letters in the name. No returned value.

public class NameReportingService
{
    public void Report(string name)
    {
        myDatabase.AddInitial(name.Substring(0,1);
        myDatabase.AddLetterCount(name.Length);
    }
}

// e.g.
myNameReportingService.Report("Bob");

How would you test this method, if not by checking that for input "Bob", a mocked database dependency would've received "B" on its AddInitial method and 3 on its AddLetterCount method?

Now you might argue that if the method also returns the same values in a result object, that you can just test this result object instead of the mocked dependency. But then you are blindly assuming that the method ensures that it posts the same values to the database that it also returns to the caller, and testing strategies are the antithesis of blind assumption.

Unit vs integration testing

tl;dr: Is it a unit test or an integration test, to ensure that the inputs to a mocked dependency were correct?

The mocked nature of the dependency directly implies that this is a unit test. If you were using a real dependency, then it'd be an integration test.

This is the very definition of unit and integration tests. A unit test only has one real component (i.e. the unit itself) where everything else is mocked, and an integration test has at least more than one real component to it. "Integration" specifically focuses on the interaction between components, which inherently means you need more than one of them.

Unit OR integration testing?

Discussing the above with a colleague, I was told that the latter is an integration test, and if I've made a mistake with the parameters to SuperWeather I should let the person doing integration tests find this out.

The goal of unit tests is being able to spot at a glance which unit failed. But your colleague's proposed test would make it impossible to spot the difference between:

  • The real weather component not yielding the correct output for the given input
  • The ReturnsTemperature method passing the wrong input to the real weather component

Without a unit test confirming that ReturnsTemperature calls its dependency correctly, you cannot figure out the difference. But with that unit test, if it passes, then you know that the issue lies with the weather component.


In essence, the issue here boils down to a question that is asked (in many subtle forms) here all the time: why bother writing unit tests, if integration tests already prove that my code is working?

The short answer here is that you don't write tests to be informative when they pass, you write them to be informative when they fail. It's precisely when test failures are encountered that you need the kind of information that localizes the source of the issue.

And when no failure is encountered, we assume that this means the code is working. I say assume, because tested codebases are not 100% bugfree all the time, it's just that the remaining bugs weren't already being tested for.

By themselves, integration tests are incapable of localizing the source of an issue. A failure in an integration test only reveals that "one of these real components failed". Which one? Why?
There might be particular assert failures that inherently point to a specific component, but that is a situational windfall and not something you can expect or rely on.

The basic issue with relying on integration tests as your sole source of testing is that while it will indeed indicate that something isn't working, it's incapable of telling you precisely what isn't working, which means you need to inspect the entire call stack and step through it to find what is going wrong.

By themselves, unit tests are incapable of confirming that the combined application works as expected (i.e. the composition of the individual components that the application is made up of). Unit tests tests individual parts, not how these parts are used as a cog in the larger machine.

Using a simple example, take a screw and a bolt. You can unit test the screw to your heart's content and find no fault in it, it's a fully working screw. You can do the same for the bolt and conclude that it is a fully working bolt.
But if one of them has a reverse thread and the other doesn't, you cannot use them together. Individually, they are valid components, they just don't quite work together.

It's an oversimplified example, but it shows the basic premise of why you need integration testing on top of unit testing.

In conclusion, unit tests are only superfluous if you always write correct integration tests, all needed integration tests, and they never ever fail. But if you hold yourself to that (ridiculous) standard, you'd be writing 100% perfect code and all of your testing effort would have been "wasted" in that sense. It's a nonsensical standard built on the presumption that your code will pass and that you therefore won't need to debug or step through the code, which is the same kind of presumption that is made by people who advocate against testing as a whole, albeit less extreme.

In any real world standard, failures are common and are precisely why we write tests so we can easily spot the failures. By skipping out on unit testing, you do a half-assed testing job. You have some confirmation that the code works or doesn't, but you haven't done anything to help yourself in cases where it doesn't. And let's not pretend like we never have any issues in our code during the development phase.

0

Since most people think "latitude and longitude" and not "longitude and latitude" you are correct in wanting to cover this with a unit test. It is unexpected and therefore prone to error. While you could wait until integration testing to figure this out, a unit test is faster and closer to the root of the problem.

Imagine it is January and you want the weather report for Bismarck, North Dakota. Instead of a blizzard warning you get a typhoon warning. Is anyone going to immediately think, "oh crap, pass longitude then latitude!"

Nope. Everyone on the team is going to have a good laugh that simmers down to a very, very confused "WTF..." It could take many painstaking hours of debugging to find the solution, because the parameter order is unexpected.

This is absolutely what unit tests are for and it is very appropriate to make an assertion about the mock inputs. I would, however, put this in its own test. Name the test accordingly. Add comments to the test explaining why such an assertion is useful. The other unit tests for the system under test do not need to concern themselves with this detail.

Then comment the call to the method with the unexpected parameter order.

This is still not perfect. The API provider could decide to reverse the parameter order, write a big helpful blog post with a giant red warning. Your unit test will still pass after upgrading, and poor Bismarck, North Dakota will still get a typhoon warning. You still need the integration test. And this one strange unit test. You need them both.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.