2

In most descriptions of unit testing as a methodology there's an idea of unit tests being as independent of implementation as possible. This is easy to understand and implement in cases when code does some actual computing, and may be represented as a math formula to certain degree. When input is correlated to output.

But assume we have code which deals with IO, or databases, hardware API or network requests. Those calls tend not to have that direct correlation, so unit testing degrades to series of EXPECT_CALL, since the only thing we may check is that calls are made with right arguments and in right order. This destroys the very essence of unit testing, since slightest change in the code forces us to update unit tests, so there's no persistent snapshot of a logic.

Even if that type of code is moved out to certain IO module, that IO module still has to be tested. And if app mostly works with db/io/etc., unit tests seem to become unusable in a project.

Is that still testable any reasonable way?

6
  • Does this answer your question? When is unit testing inappropriate or unnecessary?
    – gnat
    Jun 4 at 22:11
  • Those calls tend not to have that direct correlation Can you elaborate on this?
    – Laiv
    Jun 5 at 6:17
  • @Laiv E.g. series of calls which do get/set operation on hardware registers. There's no correlation between input and outputs, since actual results depend on hardware state, underlaying api state, etc. So in that case one can only mock apis, and check if all of the calls are done with right args, and simulate outputs. Which results in extrimely tight bond between code and tests and kind of disables tdd, since the tests are not abstract to implementation.
    – Roman
    Jun 5 at 10:38
  • There's no correlation between input and outputs Can you give an example? network traffic, db connections or calls to hardware APIs don't happen by spontaneous generation. Can you give a real-world example of this kind of code/situation you think can't be tested? I get the feeling you are making a wrong assumption in your reasoning about testing, but I don't dare to say what's the misconception. IO-bound code is perfectly testable if it's designed to be testable. Design matters.
    – Laiv
    Jun 5 at 11:04

3 Answers 3

7

Tests are not limited to verifying application logic. If all your test does is verify calls to mocks, then you are verifying implementation. You are right to question the value of such tests for the very reasons you mention in your question. This happens at the boundaries between your application and some other system, like a database or web API. There tends to be very little logic to test. That doesn't mean tests are not valuable, though.

Instead of using mocks, use the real thing — an actual database or web service. While testing terminology isn't standardized, you'll hear many people call this a form of integration testing. You are still testing the output, but not the output of a function in your program. You are testing the output generated by two or more systems working together.

These kinds of tests run slower than what you classify as a unit test. That's ok. Just don't run these tests as often. Be careful when running integration tests against a shared resource, so one test does not interfere with the output of another. File paths, database connection strings and URLs to web service should be parameterized so you can run tests against a local database, temp files, or a local copy of a web service.

That doesn't mean you cannot run tests against a shared resource. This might be desirable in a CI/CD build, but this becomes a matter of configuration and deployment.

At some point there is no benefit to mocking function calls. You need to test the real thing.

6
  • Could you rephrase the first sentence? I'm not really getting it. You suggest to just use local db/web server/hardware emulator to get that data? I was thinking of that, but that doesn't exactly line up with making the code as modular as possible for tests. So I parametrize paths, I use that fake resources, but I'm testing against multi-layer structure (e.g. facade for api -> actual api -> hardware emulator) instead of removing all those layers and plugging the hole with mock.
    – Roman
    Jun 6 at 0:50
  • I've actually tried to do something like that. I've mocked lowest level api I had, which solved the issue of non-reusable unit tests to some extent, but the level of mock detalization was absolutely insane. I had to do dozens of mock calls of low level apis, to simulate responces from underlaying system.
    – Roman
    Jun 6 at 0:53
  • @Roman: I rewrote and expanded the first sentence. Does that help? Jun 7 at 21:08
  • thanks, it helped to understand what you meant, but it kind of raises new question. I've been thinking of actually having subsystem emulation involved at the boundary, but isn't it straight against the ideology of unit testing? aka isolate single layer of code and mock the rest?
    – Roman
    Jun 7 at 22:42
  • I think my answer is: you can only write unit tests for so much of your code. At some point it's just easier and more beneficial to stop mocking and use the real thing. Jun 7 at 23:35
5

In most descriptions of unit testing as a methodology there's an idea of unit tests being as independent of implementation as possible. This is easy to understand and implement in cases when code does some actual computing, and may be represented as a math formula to certain degree. When input is correlated to output.

You've mixed together two different issues. What keeps tests "as independent of implementation as possible" is driving the code under test from a consistent level of abstraction. That leaves the code free to be refactored below that abstraction without breaking tests.

But what keeps "input correlated to output" is testing deterministic code. For example, if you're going to test a random number generator then control the seed so you know exactly what it's supposed to do. Getting code to be deterministic is all about nailing down what it's behavior depends on.

But assume we have code which deals with IO, or databases, hardware API or network requests. Those calls tend not to have that direct correlation, so unit testing degrades to series of EXPECT_CALL, since the only thing we may check is that calls are made with right arguments and in right order. This destroys the very essence of unit testing, since slightest change in the code forces us to update unit tests, so there's no persistent snapshot of a logic.

Hold on. Deterministic behavior is not limited to return result code. You most certainly can call output ports deterministically and send predictable arguments to them. What a live, shared, uncontrolled, database might do with those calls is not so predictable. Even the file system might delete that file that you just proved existed before writing to it.

Things aren't going weird here because you're testing something besides return. They're going weird because you're allowing uncontrolled behavior into the test.

Even if that type of code is moved out to certain IO module, that IO module still has to be tested. And if app mostly works with db/io/etc., unit tests seem to become unusable in a project.

Say a DB has some stored procedure. You want it under test. OK fine. Both an end-to-end test and a minimal test that each include this actual DB are possible. But only if the DB can be isolated and its state controlled enough to make the results of the test predictable.

That is absolutely not the same test as one that only confirms the DB was sent the correct call. You can mock the DB and do that test as well (it just won't include the stored procedures behavior). And you still have to control anything that would change the behavior that leads to that call.

The problem is, the more things involved in the test the easier it is for some hidden uncontrolled thing to start affecting behavior. My nickname for that is magic. I work hard to squeeze the magic out of my tests.

Keeping tests independent of implementation is a good thing but it won't squeeze the magic out. Getting control of everything the code's behavior depends on will.

3

In most descriptions of unit testing as a methodology there's an idea of unit tests being as independent of implementation as possible.

While I'm not sure if this is the core of your current question, look up black box testing vs white box testing. The former intentionally assumes no knowledge of the internals, whereas the latter permits it.

This destroys the very essence of unit testing, since slightest change in the code forces us to update unit tests

Very much make a distinction here between needing to refactor test code so that it compiles, and needing to revisit test cases because the behavior has changed.

When you make intentional changes to the behavior, it is inevitable that you need to revisit your test cases. You may need to add new tests, deprecate now obsolete ones, or adjust existing cases to match the new behavior.

When you make intentional changes to the interface, but not the underlying behavior, then it is inevitable that you have to update your test code to work with this new interface. However, the test cases themselves don't need to be updated since the behavior remains the same.

Unit testing degrades to series of EXPECT_CALL, since the only thing we may check is that calls are made with right arguments and in right order. This destroys the very essence of unit testing

I very much disagree. It is the very essence of unit tests that you only look at the unit and mock everything outside of it.

The things that you have excluded from your unit test (i.e. your external dependencies) are not under test, and therefore their implementation should not have an impact on this test's results.

When you cannot account for the actual implementation, the only remaining verification is to make sure that the interface is being called correctly, which is what you're doing with your EXPECT_CALL assertions.

so there's no persistent snapshot of a logic

I don't understand what you mean here. A snapshot of the logic would be a commit history in your versioning system, including the tests in that history which can at all times be re-run if their outcome was not already recorded.

If, instead, you are worried that a refactor of your code (without an adjustment of your test case) is going to lead your developers to accidentally make changes to your test cases anyway, then you have a different fish to fry. Whether this is low developer skill, unreasonably high test code complexity, or just a mistrust of your developers' abilities; I cannot judge this based on the question as asked.

Even if that type of code is moved out to certain IO module, that IO module still has to be tested.

I'm not sure what your argument here is. The core of unit testing is that each unit has its own dedicated tests. So yeah, a module would have tests written for that module - assuming there is any meaningful behavior worth testing, which might not be the case for IO specifically.

And if app mostly works with db/io/etc., unit tests seem to become unusable in a project.

If your application straight up serves the data from the data source without any real alteration or logic on top of it; then there's not much behavior to test. On the low end of the spectrum, you'd not be writing much more than a mapping test (which I don't considered relevant to test but opinions vary here).

Whether or not your unit tests have purpose directly correlates to how much behavior there is to meaningfully test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.