Misguided Unit Testing Approach

Question

My organization is investing time in writing the wrong kinds of tests (unit tests that are tied far too heavily to the implementation, are very time consuming to create, and often make the system harder to change). Also, context is often thrown away and tests are skewed to improve coverage metrics.

The same assumptions that are baked into the code are baked into the unit tests.

I believe that code coverage is a misleading metric. One can have 100% code coverage and still have many bugs, many untested edge cases, etc. The costliness of how much work it takes to get the last 10% of coverage does not provide commensure value.

The tests do not serve well as "readable specs" (an ideal that sounds great on paper but often does not translate).

I found a test that, for example, asserted that a certain dropdown menu "contains 7 items." That doesn't map to any real-world expectation held by business users or even technical people. It seems like the author of that test was just looking for "something to assert."

I fear that we are measuring what is easy to measure over what is important (though perhaps less measurable).

Some of the tests are written at such a level that they set up mocks to verify every tiny step of the implementation (was Dispose called on the UnitOfWork)? Etc.

I would prefer outcome-based testing - verify that the outcome is right, the "what" and not the "how."

I think it would be great to do BDD or ATDD (acceptance test driven development) but organizationally we are not ready for a big paradigm shift to Cucumber or SpecFlow or a similar tool (although maybe it is a good long term goal).

Here's the problem. If you express a desire to write less tests, the testing zealous will look at you as though you have revealed yourself as deeply ignorant/immoral. Maybe I'm projecting a little bit? But no one wants to sound like they're suggesting we "cut corners" for speed. Nonetheless, I think our process is dragged down by blind adherence to protocols that aren't always contextually appropriate.

Is there a "post-zealotry" stance on testing that I can start shifting us toward?

Answer 1

No, you are not crazy. TDD/unit tests are great, but it must be applied correctly.

There is a phenomenon called cargo-cult coding/architecture. That is when you apply an approach without knowing why you are doing so. This seems to have been the case with automated tests on your project.

To me,

• Automated Integration Tests have uses, however, focus on unit tests should be preferred when ever possible.
• Code based Automated Testing the UI is a waste of time. (I am not referring to a test framework like Selenium, which can be useful)
• Any test that is more complex than the code it is testing is just another potential point of failure, and therefore, at best, a waste of time

The type of thing you are referring to happening on your project is a symptom of the following:

• Tightly coupled code
• Failure to isolate business logic from the infrastructure components of the code base

If you code was less coupled, only a minimum of mocking would be necessary.

If your business logic was properly isolated, it would be very obvious as to how to write effective and productive unit tests of your code base.