7

I'm relatively new to TDD and have been thinking a lot about how to manage the perpetually growing pool of tests that comes with it.

One of my biggest concerns is about false positives.

In my experience, it's not uncommon at all for a feature to undergo fairly massive changes over time. To implement a change we would first write a test (or many tests) for the new behaviour and then code the change. Ideally, all the tests would pass once the feature change has been correctly implemented.

But what about the tests that covered the first incarnation of the feature? There is no guarantee that:

  1. they are still relevant;
  2. they still pass;
  3. their pass/fail status is even correct.

Reviewing/updating/deleting these tests in a relatively small pool of tests may be manageable, but what about when you have thousands or tens of thousands of tests? It seems to me that there is a big risk that your collection of tests will eventually contain many false positives. And, considering that these tests are a form of system documentation, this is pretty bad!

Question: How do you mitigate the risk of accumulating False Positive tests when applying TDD or BDD (or anything that eventually leads to having a massive collection of tests)?

11

The key is to have all your tests clearly refer to the precise bit of functionality they are supposed to test.

It can be via very expressive test names and strict one-assert-per-test discipline, or really clever subdirectory structure in your test suite, or references from test suite to source code, or whatever. The point is that you can quickly identify which tests will become obsolete.

Then you delete them.

That sounds foolhardy, but every major change deletes some functionality and replaces it with new, different functionality. If you're using TDD, you're going to write new tests for every bit of that anyway, and fixing old tests instead of forging ahead with new tests and production code is just a waste of time. And this is why questions of organization, coupling, coherence etc. etc. are just as important in test code as in production code!

  • Ok, so this hinges on having a sufficiently reliable mechanism for linking coded tests to the functionality they are supposed to test? While the example mechanisms you've mentioned seem sufficient (and I've seen some already), they also seem prone to human error. I guess I'm trying to say that I'm kind of surprised such an important aspect of managing tests is left up to establishing a convention and then manually enforcing it. – MetaFight Dec 17 '13 at 11:48
  • 1
    I admit I felt somewhat surprised, too, when I wrote that, but I honestly don't know any better. I suppose the limiting case, when you have no mechanism for cross-referencing, is simply to forge ahead and later detect the related tests by the fact that they fail - not a good solution, but reliable by definition. – Kilian Foth Dec 17 '13 at 11:51
  • But if those tests never fail then, at best, they provide no value, and at worst, they are misinformation. By misinformation, I mean that anyone trying to read the tests as a form of documentation will be misled. But as you mentioned, this is the limiting case when there is no mechanism for cross-referencing. – MetaFight Dec 17 '13 at 12:49
  • 1
    Writing tests well is very hard This is part of it; making sure your tests completely describe the desired behaviour so that changes to the behaviour that are undesirable result in a test fail. – Dancrumb Dec 17 '13 at 22:02
  • 1
    @Dancrumb I think your last comment is spot on. I'd argue that sometimes, however, the effort required to make tests fail when they should fail is too high. In those cases, having a mechanism (outside of the test itself) which draws the maintainer's attention to the test would be a happy compromise. – MetaFight Dec 17 '13 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.