Unit testing code which does not establish correlation between input and output

Question

In most descriptions of unit testing as a methodology there's an idea of unit tests being as independent of implementation as possible. This is easy to understand and implement in cases when code does some actual computing, and may be represented as a math formula to certain degree. When input is correlated to output.

But assume we have code which deals with IO, or databases, hardware API or network requests. Those calls tend not to have that direct correlation, so unit testing degrades to series of EXPECT_CALL, since the only thing we may check is that calls are made with right arguments and in right order. This destroys the very essence of unit testing, since slightest change in the code forces us to update unit tests, so there's no persistent snapshot of a logic.

Even if that type of code is moved out to certain IO module, that IO module still has to be tested. And if app mostly works with db/io/etc., unit tests seem to become unusable in a project.

Is that still testable any reasonable way?

Answer 1

Tests are not limited to verifying application logic. If all your test does is verify calls to mocks, then you are verifying implementation. You are right to question the value of such tests for the very reasons you mention in your question. This happens at the boundaries between your application and some other system, like a database or web API. There tends to be very little logic to test. That doesn't mean tests are not valuable, though.

Instead of using mocks, use the real thing — an actual database or web service. While testing terminology isn't standardized, you'll hear many people call this a form of integration testing. You are still testing the output, but not the output of a function in your program. You are testing the output generated by two or more systems working together.

These kinds of tests run slower than what you classify as a unit test. That's ok. Just don't run these tests as often. Be careful when running integration tests against a shared resource, so one test does not interfere with the output of another. File paths, database connection strings and URLs to web service should be parameterized so you can run tests against a local database, temp files, or a local copy of a web service.

That doesn't mean you cannot run tests against a shared resource. This might be desirable in a CI/CD build, but this becomes a matter of configuration and deployment.

At some point there is no benefit to mocking function calls. You need to test the real thing.

Answer 2

In most descriptions of unit testing as a methodology there's an idea of unit tests being as independent of implementation as possible. This is easy to understand and implement in cases when code does some actual computing, and may be represented as a math formula to certain degree. When input is correlated to output.

You've mixed together two different issues. What keeps tests "as independent of implementation as possible" is driving the code under test from a consistent level of abstraction. That leaves the code free to be refactored below that abstraction without breaking tests.

But what keeps "input correlated to output" is testing deterministic code. For example, if you're going to test a random number generator then control the seed so you know exactly what it's supposed to do. Getting code to be deterministic is all about nailing down what it's behavior depends on.

But assume we have code which deals with IO, or databases, hardware API or network requests. Those calls tend not to have that direct correlation, so unit testing degrades to series of EXPECT_CALL, since the only thing we may check is that calls are made with right arguments and in right order. This destroys the very essence of unit testing, since slightest change in the code forces us to update unit tests, so there's no persistent snapshot of a logic.

Hold on. Deterministic behavior is not limited to return result code. You most certainly can call output ports deterministically and send predictable arguments to them. What a live, shared, uncontrolled, database might do with those calls is not so predictable. Even the file system might delete that file that you just proved existed before writing to it.

Things aren't going weird here because you're testing something besides return. They're going weird because you're allowing uncontrolled behavior into the test.

Even if that type of code is moved out to certain IO module, that IO module still has to be tested. And if app mostly works with db/io/etc., unit tests seem to become unusable in a project.

Say a DB has some stored procedure. You want it under test. OK fine. Both an end-to-end test and a minimal test that each include this actual DB are possible. But only if the DB can be isolated and its state controlled enough to make the results of the test predictable.

That is absolutely not the same test as one that only confirms the DB was sent the correct call. You can mock the DB and do that test as well (it just won't include the stored procedures behavior). And you still have to control anything that would change the behavior that leads to that call.

The problem is, the more things involved in the test the easier it is for some hidden uncontrolled thing to start affecting behavior. My nickname for that is magic. I work hard to squeeze the magic out of my tests.

Keeping tests independent of implementation is a good thing but it won't squeeze the magic out. Getting control of everything the code's behavior depends on will.

Answer 3

In most descriptions of unit testing as a methodology there's an idea of unit tests being as independent of implementation as possible.

While I'm not sure if this is the core of your current question, look up black box testing vs white box testing. The former intentionally assumes no knowledge of the internals, whereas the latter permits it.

This destroys the very essence of unit testing, since slightest change in the code forces us to update unit tests

Very much make a distinction here between needing to refactor test code so that it compiles, and needing to revisit test cases because the behavior has changed.

When you make intentional changes to the behavior, it is inevitable that you need to revisit your test cases. You may need to add new tests, deprecate now obsolete ones, or adjust existing cases to match the new behavior.

When you make intentional changes to the interface, but not the underlying behavior, then it is inevitable that you have to update your test code to work with this new interface. However, the test cases themselves don't need to be updated since the behavior remains the same.

Unit testing degrades to series of EXPECT_CALL, since the only thing we may check is that calls are made with right arguments and in right order. This destroys the very essence of unit testing

I very much disagree. It is the very essence of unit tests that you only look at the unit and mock everything outside of it.

The things that you have excluded from your unit test (i.e. your external dependencies) are not under test, and therefore their implementation should not have an impact on this test's results.

When you cannot account for the actual implementation, the only remaining verification is to make sure that the interface is being called correctly, which is what you're doing with your EXPECT_CALL assertions.

so there's no persistent snapshot of a logic

I don't understand what you mean here. A snapshot of the logic would be a commit history in your versioning system, including the tests in that history which can at all times be re-run if their outcome was not already recorded.

If, instead, you are worried that a refactor of your code (without an adjustment of your test case) is going to lead your developers to accidentally make changes to your test cases anyway, then you have a different fish to fry. Whether this is low developer skill, unreasonably high test code complexity, or just a mistrust of your developers' abilities; I cannot judge this based on the question as asked.

Even if that type of code is moved out to certain IO module, that IO module still has to be tested.

I'm not sure what your argument here is. The core of unit testing is that each unit has its own dedicated tests. So yeah, a module would have tests written for that module - assuming there is any meaningful behavior worth testing, which might not be the case for IO specifically.

And if app mostly works with db/io/etc., unit tests seem to become unusable in a project.

If your application straight up serves the data from the data source without any real alteration or logic on top of it; then there's not much behavior to test. On the low end of the spectrum, you'd not be writing much more than a mapping test (which I don't considered relevant to test but opinions vary here).

Whether or not your unit tests have purpose directly correlates to how much behavior there is to meaningfully test.