Types of unit tests based on usefulness

Question

From value point of view I see two groups of unit tests in my practice:

1. Tests that test some non-trivial logic. Writing them (either before implementation or after) reveals some problems/potential bugs and helps to be confident in case logic is changed in the future.
2. Tests that test some very trivial logic. Those tests more like document code (typically with mocks) than test it. The maintenance workflow of those tests is not "some logic changed, test became red - thanks God I wrote this test" but "some trivial code changed, test became false negative - I have to maintain (rewrite) the test without gaining any profit". Most of the time those tests are not worth maintaining (except religious reasons). And according to my experience in many systems those tests are like 80% of all tests.

I am trying to find out what other guys think on the topic of unit tests separation by value and how it corresponds to my separation. But what I mostly see is either fulltime TDD propaganda or tests-are-useless-just-write-the-code propaganda. I'm interested in something in the middle. Your own thoughts or references to articles/papers/books are welcome.

Answer 1

I think it's natural to encounter a divide within unit testing. There are many different opinions on how to do it properly and naturally all other opinions are inherently wrong. There are quite a few articles on DrDobbs recently that explore this very issue to which I link at the end of my answer.

The first problem I see with tests is that it's easy to get them wrong. In my college C++ class we were exposed to unit tests in both the first and second semester. We knew nothing about programming in general in either of the semesters- we were trying to learn the fundamentals of programming via C++. Now imagine telling the students, "Oh hey, you wrote a little yearly tax calculator! Now write some unit tests to ensure that it works correctly." The results should be obvious- they were all horrible, including my attempts.

Once you admit that you suck at writing unit tests and wish to get better, you will soon be faced with either trendy styles of testing or different methodologies. By testing methodologies I refer to practices such as test-first or what Andrew Binstock of DrDobbs does, which is write the tests alongside the code. Both have their pros and cons and I refuse to go into any subjective detail because that will incite a flame war. If you're not confused about which programming methodology is better, then maybe the style of testing will do the trick. Should you use TDD, BDD, Property-based testing? JUnit has advanced concepts called Theories that blurs the line between TDD and Property-based testing. Which to use when?

tl;dr It's easy to get testing wrong, it's incredibly opinionated and I don't believe that any one testing methodology is inherently better as long as they are used diligently and professionally within the context that they're appropriate in. Furthermore, testing is in my mind an extension to assertions or sanity-tests that used to ensure a fail-fast ad-hoc approach to development which is now much, much easier.

For a subjective opinion, I prefer to write "phases" of tests, for lack of a better phrase. I write unit tests which test classes in isolation, using mocks where necessary. These will probably be executed with JUnit or something similar. Then I write integration or acceptance tests, these are run separately and usually only a few times a day. These are your non-trivial use case. I usually use BDD as it is nice to express features in natural language, something that JUnit cannot easily provide.

Lastly, resources. These will present conflicting opinions mostly centered around unit testing in different languages and with different frameworks. They should present the divide in ideology and methodology while allowing you to make up your own opinion as long as I haven't manipulated yours too much already :)

[1] The Corruption of Agile by Andrew Binstock

[2] Response to the Responses of the previous article

[3] Response to Corruption of Agile by Uncle Bob

[4]Response to Corruption of Agile by Rob Myers

[5]Why Bother With Cucumber Testing?

[6] You're Cuking it Wrong

[7]Step Away From the Tools

[8]Commentary on 'Roman Numerals Kata with Commentary'

[9]Roman Numerals Kata With Commentary

Answer 2

I believe it is important to have tests of both types and to use them where appropriate.

Like you said, there are two extremes and I honestly do not agree with either one as well.

The key is that unit tests have to cover business rules and requirements. If there is a requirement that the system must track a person's age, write "trivial" tests to ensure the age is a nonnegative integer. You are testing the domain of data required by the system: while trivial, it has value because it is enforcing the parameters of the system.

Likewise with more complex tests, they have to bring value. Sure, you can write a test that validates something that is not a requirement but should be enforced in an ivory tower somewhere, but that is time better spent writing tests that validate the requirements for which the customer is paying you. For example, why write a test that validates your code can deal with an input stream that times out, when the only streams are from local files, not the network?

I firmly believe in unit testing and use TDD wherever it makes sense. Unit tests certainly bring value in the form of increased quality and "fail fast" behavior when changing code. However, there is the old 80/20 rule to keep in mind too. At some point you will reach diminishing returns when writing tests, and you need to move to more productive work even if there is some measurable value to be had from writing more tests.

Answer 3

Here's my take on it: all tests have costs:

• initial time and effort:
  • think about what to test and how to test it
  • implement the test and make sure it's testing what it's supposed to
• ongoing maintenance
  • making sure the test is still doing what it's supposed to do as the code evolves naturally
• running the test
  • execution time
  • analyzing the results

We also intend for all tests to provide benefits (and in my experience, nearly all tests do provide benefits):

• specification
• highlight corner cases
• prevent regression
• automatic verification
• examples of API usage
• quantification of specific properties (time, space)

So it's pretty easy to see that if you write a bunch of tests, they're probably going to have some value. Where this gets complicated is when you start comparing that value (which, by the way, you may not know in advance -- if you throw away your code, the regression tests lose their value) to the cost.

Now, your time and effort are limited. You would like to choose to do those things that provide the most benefit at the least cost. And I think that is a very difficult thing to do, not least because it may require knowledge that one does not have or would be expensive to obtain.

And that's the real rub between these different approaches. I believe they've all identified testing strategies which are beneficial. However, each strategy has different costs and benefits in general. Also, the costs and benefits of each strategy will probably be heavily dependent on the specifics of the project, the domain, and the team. In other words, there may be multiple best answers.

In some cases, pumping out code without tests may provide the best benefits/costs. In other cases, a thorough test suite may be better. In still other cases, improving the design may be the best thing to do.

Answer 4

What is a unit test, really? And is there really such a big dichotomy in play here?

We work in a field where reading literally one bit past the end of a buffer can totally crash a program, or cause it to produce a totally inaccurate result, or as evidenced by the recent "HeartBleed" TLS bug, lay a supposedly secure system wide open without producing any direct evidence of the flaw.

It is impossible to eliminate all complexity from these systems. But our job is, to the extent possible, to minimize and manage that complexity.

Is a unit test a test that confirms, for example, that a reservation is successfully posted in three different systems, a log entry is created and an Email confirmation is sent out?

I'm going to say no. That's an integration test. And those most definitely have their place, but they're also a different topic.

An integration test works to confirm the overall function of an entire "feature." But the code behind that feature should be broken down into simple, testable building blocks, aka "units."

So a unit test should have a very limited scope.

Which implies that the code tested by the unit test should have a very limited scope.

Which further implies that one of the pillars of good design is to break your complex problem down into smaller, single-purpose pieces (to the extent possible) which can be tested in relative isolation from one another.

What you end up with is a system made out of reliable foundation components, and you know if any of those foundational units of code break because you have written simple, small, limited-scope tests to tell you exactly that.

In many cases you should also probably have multiple tests per unit. The tests themselves should be simple, testing one and only one behavior to the extent possible.

The notion of a "unit test" testing non-trivial, elaborate, complex logic is, I think, a bit of an oxymoron.

So if that kind of deliberate design break-down has taken place, then how in the world could a unit test suddenly start producing false positives, unless the basic function of the tested code unit has changed? And if that has happened, then you better believe there are some non-obvious ripple effects in play. Your broken test, the one that seems to be producing a false positive, is actually warning you that some change has broken a wider circle of dependencies in the code base, and it needs to be examined and fixed.

Some of those units (many of them) may need to be tested by using mock objects, but that doesn't mean you have to write more complex or elaborate tests.

Going back to my contrived example of a reservation system, you really can't be sending requests off to a live reservation database or third-party service (or even a "dev" instance of it) every time you unit test your code.

So you use mocks which present the same interface contract. The tests can then validate the behavior of a relatively small, deterministic chunk of code. Green all down the board then tells you that the blocks that comprise your foundation aren't broken.

But the logic of the individual unit tests themselves remains as simple as possible.

Answer 5

This is of course, just my oppinion, but having spent the last few months learning functional programming in fsharp (coming from a C# background) has made me realize a few things.

As the OP stated, there are typically 2 types of "unit tests" we see day to day. Tests that cover the in's and out's of a method, which are generally the most valuable, but are hard to do for 80% of the system which is less about "algorithims" and more about "abstractions".

The other type, is testing abstraction interactivity, typically involve mocking. In my oppinion, this tests are mostly necessary due to the design of your application. Ommiting them, and you risk weird bugs and spagetti code, because people don't think about their design properly unless they are forced to do the tests first (and even then, usually mess it up). The problem is not so much the testing methodology, but the underlying design of the system. Most systems built with imperative or OO languages have an inhert reliance on "side effects" aka "Do this, but don't tell me anything". When you rely on the side effect, you need to test it, because a business requirement or operation is usually a part of it.

When you design your system in a more functional way, where you avoid building dependencies on side-effects and avoid state changes/tracking through immutability, it enables you to focus more heavily on "ins and outs" tests, which clearly test more the action, and less how you get there. You will be amazed what things like immutability can give you in terms of much simpler solutions to the same problems, and when you are no longer dependent on "side-effects" you can do things like parallelizing and asynchronous programming with almost no additional cost.

Since I started coding in Fsharp, I have not needed a mocking framework for anything, and have even shed my dependency altogether on an IOC Container. My tests are driven by business need and value, and not on heavy abstraction layers typically needed to achieve composition in imperative programming.