5

I was reading some questions about TDD here and came up with my own.

I understand TDD to be a way of developing or capturing the detailed, internal specification/requirements for code in the course of writing that code.

We proceed step by step and add falsifiable propositions to the specification body in the form of tests. We show these to be false and then write code to make them true. Then refactor while keeping them true.

The problem is, how do we know that all the existing tests are staying falsifiable (with regard to their original feature)?

In the course of adding some new element to the code via this process, suppose that some tests can easily become outdated in the following sense: those tests were initially associated with some specific change, and if that change is now reverted in the current baseline, perhaps they still pass. Whatever is making them pass now is not that original feature which flipped them from fail to pass.

Those tests/specifications seem like garbage which adds bulk to the specification base without any substance. Or is there still value in a test which doesn't go false if the original code change which made it true is reverted, and it's not obvious what other code change to the current base would make it go false?

Is there some TDD process to identify these tests? And are they then removed or rewritten or what?

Or, alternatively, is this entirely prevented somehow?

  • Perhaps I'm missing something here, but don't existing tests become candidates for review when they go red during a refactor? At that point, you can decide whether you broke the test, or whether the test's assumptions no longer apply. – Robert Harvey Jul 28 '17 at 23:08
  • @RobertHarvey Right; but this is about tests that don't go red. What if they can't go red any more? Or at least, not by reverting the original change which was introduced to flip them originally from red to green. – Kaz Jul 28 '17 at 23:12
  • Why would that be a problem? In your question, you apparently see those tests as "dead code," but unless you're willing to hold a periodic review to challenge your test assumptions, I don't see how it's a significant factor in the ongoing evolution of an application. As with anything else in software development, you have to weight the benefits against the costs, and removing dead code in unit tests that have no negative effect beyond code bulk seems pretty far down the totem pole to me. – Robert Harvey Jul 28 '17 at 23:14
  • What happens if the current code makes it so the test cannot go red. (There are sufficient validations to ensure the failure case does not occur.) It is still possible one of those validation will be removed or broken. The other case would be removed functionality, but that should cause the test to go red. – BillThor Jul 28 '17 at 23:35
  • 2
    Refactoring tests against a red bar is relevant here. – RubberDuck Jul 29 '17 at 1:43
4

One of two things is wrong here.

You did not really follow TDD

If you follow TDD, then you write a failing test, then write code to make the test pass. If you remove the code which makes it pass, then your test ought to have broken. Perhaps you did not actually follow TDD and therefore built a self-passing test.

Your tests verify lack of a problem rather than success

Unit tests should not be designed to test problems. They should be designed as a statement of fact, which must be upheld. For example:

@Test
public void tableIsSortedAlphabetically() {
    Table t = new Table();
    t.addRow("Hello");
    t.addRow("World");
    t.addRow("Apple");
    assertEquals("Apple", t.getRow(0));
}

Notice how this is phrased. We are testing that something is sorted a particular way. If the test is designed this way and it is implemented well, there is no way you can remove the code which sorts a table alphabetically without this test failing.

Here is a test which would still pass if the code under test were removed.

@Test
public void userCanBeCreated() {
    User user = new User();
    user.setFirstName("Johnny");
    user.setLastName("Appleseed");

    try {
        user.save();
    }
    catch (Exception e) {
        fail(e);
    }
}

If the implementation of user.save() were changed to do nothing, this test would still pass because the test does not actually verify if the behavior is correct, but just tests that it doesn't fail.

  • 5
    I hate to be "That Guy", but your first test doesn't test your table is sorted alphabetically. Please note I'm not trying to pick on you, but in general I find examples of TDD to be either too simple to be useful, or just plain misleading (another example I find misleading and which tends to crop up in examples of TDD is "finding" the function add(x, y) by writing tests) – Andres F. Jul 29 '17 at 1:12
  • 1
    Sure, it should have an assertion for all 3 elements. And create several Table objects to verify it does not happen to coincidentally sort correctly for this one data set. – Brandon Jul 31 '17 at 17:28
  • 1
    Even better, it could be a property test ala QuickCheck/ScalaCheck, and test random tables of random length to verify they are alphabetically sorted, providing a counterexample when the property gets broken. – Andres F. Jul 31 '17 at 19:41
3

The problem is, how do we know that all the existing tests are staying falsifiable (with regard to their original feature)?

One answer is to make the tests immutable

The basic pattern of TDD is this: you implement a test, see it fail, update the production code, see it pass. So long as the test stays unchanged from this point, you can be confident that the current implementation continues to satisfy that constraint.

Those tests/specifications seem like garbage which adds bulk to the specification base without any substance. Or is there still value in a test which doesn't go false if the original code change which made it true is reverted

Yes, because the test itself is still documentation of the specification. A good test doesn't normally force a specific implementation -- if that were the case, refactoring wouldn't work. Instead, the test only constrains the set of possible implementations to those that produce a specific result.

Is there some TDD process to identify these tests?

Not really. There are some protocols that you can apply to try to identify tests that are no longer constraining anything -- mutation testing, for instance.

And are they then removed or rewritten or what?

If the specification has changed, then you can retire a test. If the specification is unchanged, but you've lost confidence in the test, then introduce a new test, establish confidence in it, and then retire the old.

1

Tests are aimed at behavior. Not lines of code.

Code coverage is nice but if I can get what I want with no code at all it sounds good to me.

What disturbs me is the idea of the test not being falsifiable. You shouldn't have the test if it wasn't once false. But if you add a line of code that makes test 2 pass and also makes test 1 pass that used to only pass because of other code well then it's time for that other, now useless, code to be deleted. But even without deleting it the test is still falsifiable. You just have to break a different line to code to do it.

A test might know what method it calls when testing but it doesn't know where the behavior that method exhibits is coded. So how could I possibly insist that there is a problem with a test that goes false when it should, which is when the behavior breaks, just because where the behavior is coded has changed?

Now all that said, if you can't look at the test and the code and figure out where the behavior under test is coded you have bigger problems.

Now this isn't saying that tests never need to die. You should change your tests as well.

is this entirely prevented somehow?

Just review your tests regularly. You should be doing that anyway. It you spot one that you aren't sure how it could be falsified stop everything and go break something to turn it false. This is how you test the test.

0

You shouldn't think of tests as testing specific lines of code. Rather a test verifies that an aspect of a requirement is fulfilled by the system or subsystem one way or another.

A test will not be obsolete as long as the requirement is there, even if the code which originally fulfilled the requirement is completely rewritten or replaced.

As long as a test actually touch the system, it has the potential to go red, if somebody makes a change to the code. I mean, if somebody deleted all the code, surely all your tests would fail.

  • If a test doesn't test specific lines of code, why is necessary that the initial change which flips a test from red to green must be minimal (thus almost certainly traceable to a few lines of code). After numerous refactorings, the property of the program which corresponds to that test might be spread in multiple places, yet the test has value. – Kaz Aug 1 '17 at 2:41
0

One of the TDD benefits is that you are forced to write code against abstractions. So when you recognize some logic which can be moved outside of current unit behind abstraction and passed in the unit as dependency. You end up with bunch of abstractions your unit depend on.

With abstractions you don't need change existed implementation, but just implement new one. Where later you can recognize classes not referenced by anybody, most of the IDE provide this feature, and remove them with correspond tests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.