TDD: how are obsolete tests flushed out?

Question

I was reading some questions about TDD here and came up with my own.

I understand TDD to be a way of developing or capturing the detailed, internal specification/requirements for code in the course of writing that code.

We proceed step by step and add falsifiable propositions to the specification body in the form of tests. We show these to be false and then write code to make them true. Then refactor while keeping them true.

The problem is, how do we know that all the existing tests are staying falsifiable (with regard to their original feature)?

In the course of adding some new element to the code via this process, suppose that some tests can easily become outdated in the following sense: those tests were initially associated with some specific change, and if that change is now reverted in the current baseline, perhaps they still pass. Whatever is making them pass now is not that original feature which flipped them from fail to pass.

Those tests/specifications seem like garbage which adds bulk to the specification base without any substance. Or is there still value in a test which doesn't go false if the original code change which made it true is reverted, and it's not obvious what other code change to the current base would make it go false?

Is there some TDD process to identify these tests? And are they then removed or rewritten or what?

Or, alternatively, is this entirely prevented somehow?

Answer 1

One of two things is wrong here.

You did not really follow TDD

If you follow TDD, then you write a failing test, then write code to make the test pass. If you remove the code which makes it pass, then your test ought to have broken. Perhaps you did not actually follow TDD and therefore built a self-passing test.

Your tests verify lack of a problem rather than success

Unit tests should not be designed to test problems. They should be designed as a statement of fact, which must be upheld. For example:

@Test
public void tableIsSortedAlphabetically() {
    Table t = new Table();
    t.addRow("Hello");
    t.addRow("World");
    t.addRow("Apple");
    assertEquals("Apple", t.getRow(0));
}

Notice how this is phrased. We are testing that something is sorted a particular way. If the test is designed this way and it is implemented well, there is no way you can remove the code which sorts a table alphabetically without this test failing.

Here is a test which would still pass if the code under test were removed.

@Test
public void userCanBeCreated() {
    User user = new User();
    user.setFirstName("Johnny");
    user.setLastName("Appleseed");

    try {
        user.save();
    }
    catch (Exception e) {
        fail(e);
    }
}

If the implementation of user.save() were changed to do nothing, this test would still pass because the test does not actually verify if the behavior is correct, but just tests that it doesn't fail.

Answer 2

The problem is, how do we know that all the existing tests are staying falsifiable (with regard to their original feature)?

One answer is to make the tests immutable

The basic pattern of TDD is this: you implement a test, see it fail, update the production code, see it pass. So long as the test stays unchanged from this point, you can be confident that the current implementation continues to satisfy that constraint.

Those tests/specifications seem like garbage which adds bulk to the specification base without any substance. Or is there still value in a test which doesn't go false if the original code change which made it true is reverted

Yes, because the test itself is still documentation of the specification. A good test doesn't normally force a specific implementation -- if that were the case, refactoring wouldn't work. Instead, the test only constrains the set of possible implementations to those that produce a specific result.

Is there some TDD process to identify these tests?

Not really. There are some protocols that you can apply to try to identify tests that are no longer constraining anything -- mutation testing, for instance.

And are they then removed or rewritten or what?

If the specification has changed, then you can retire a test. If the specification is unchanged, but you've lost confidence in the test, then introduce a new test, establish confidence in it, and then retire the old.

Answer 3

Tests are aimed at behavior. Not lines of code.

Code coverage is nice but if I can get what I want with no code at all it sounds good to me.

What disturbs me is the idea of the test not being falsifiable. You shouldn't have the test if it wasn't once false. But if you add a line of code that makes test 2 pass and also makes test 1 pass that used to only pass because of other code well then it's time for that other, now useless, code to be deleted. But even without deleting it the test is still falsifiable. You just have to break a different line to code to do it.

A test might know what method it calls when testing but it doesn't know where the behavior that method exhibits is coded. So how could I possibly insist that there is a problem with a test that goes false when it should, which is when the behavior breaks, just because where the behavior is coded has changed?

Now all that said, if you can't look at the test and the code and figure out where the behavior under test is coded you have bigger problems.

Now this isn't saying that tests never need to die. You should change your tests as well.

is this entirely prevented somehow?

Just review your tests regularly. You should be doing that anyway. It you spot one that you aren't sure how it could be falsified stop everything and go break something to turn it false. This is how you test the test.

Answer 4

You shouldn't think of tests as testing specific lines of code. Rather a test verifies that an aspect of a requirement is fulfilled by the system or subsystem one way or another.

A test will not be obsolete as long as the requirement is there, even if the code which originally fulfilled the requirement is completely rewritten or replaced.

As long as a test actually touch the system, it has the potential to go red, if somebody makes a change to the code. I mean, if somebody deleted all the code, surely all your tests would fail.

Answer 5

One of the TDD benefits is that you are forced to write code against abstractions. So when you recognize some logic which can be moved outside of current unit behind abstraction and passed in the unit as dependency. You end up with bunch of abstractions your unit depend on.

With abstractions you don't need change existed implementation, but just implement new one. Where later you can recognize classes not referenced by anybody, most of the IDE provide this feature, and remove them with correspond tests.