21

What is the best course of action in TDD if, after implementing the logic correctly, the test still fails (because there is a mistake in the test)?

For example, suppose you would like to develop the following function:

int add(int a, int b) {
    return a + b;
}

Suppose we develop it in the following steps:

  1. Write test (no function yet):

    // test1
    Assert.assertEquals(5, add(2, 3));
    

    Results in compilation error.

  2. Write a dummy function implementation:

    int add(int a, int b) {
        return 5;
    }
    

    Result: test1 passes.

  3. Add another test case:

    // test2 -- notice the wrong expected value (should be 11)!
    Assert.assertEquals(12, add(5, 6));
    

    Result: test2 fails, test1 still passes.

  4. Write real implementation:

    int add(int a, int b) {
        return a + b;
    }
    

    Result: test1 still passes, test2 still fails (since 11 != 12).

In this particular case: would it be better to:

  1. correct test2, and see that it now passes, or
  2. delete the new portion of implementation (i.e. go back to step #2 above), correct test2 and let it fail, and then reintroduce the correct implementation (step #4. above).

Or is there some other, cleverer way?

While I understand that the example problem is rather trivial, I'm interested in what to do in the generic case, which might be more complex than the addition of two numbers.

EDIT (In response to the answer of @Thomas Junk):

The focus of this question is what TDD suggests in such a case, not what is "the universal best practice" for achieving good code or tests (which might be different than the TDD-way).

  • 3
    Refactoring against the red bar is a relevant concept. – RubberDuck Jan 29 '17 at 2:34
  • 5
    Clearly, you need to be doing TDD on your TDD. – Blrfl Jan 29 '17 at 15:17
  • 17
    If anyone ever asks me why I am skeptic of TDD I will point them to this question. This is Kafkaesque. – Traubenfuchs Jan 29 '17 at 18:45
  • @Blrfl that's what Xibit tells us »I put the TDD in TDD so you could TDD while TDDing« :D – Thomas Junk Feb 1 '17 at 8:30
  • 3
    @Traubenfuchs I admit the question seems silly at first sight and I'm not an advocate of "do TDD all the time", but I do believe there is a strong benefit to see a test fail, then write code that makes the test pass (which is what this question actually is about, after all). – Vincent Savard Feb 1 '17 at 13:42
31

The absolutely critical thing is that you see the test both pass and fail.

Whether you delete the code to make the test fail then rewrite the code or sneak it off to the clipboard only to paste it back later doesn't matter. TDD never said you had to retype anything. It wants to know the test passes only when it should pass and fails only when it should fail.

Seeing the test both pass and fail is how you test the test. Never trust a test you've never seen do both.


Refactoring Against The Red Bar gives us formal steps for refactoring a working test:

  • Run the test
    • Note the green bar
    • Break the code being tested
  • Run the test
    • Note the red bar
    • Refactor the test
  • Run the test
    • Note the red bar
    • Un-break the code being tested
  • Run the test
    • Note the green bar

However, we aren't refactoring a working test. We have to transform a buggy test. One concern is code that was introduced while only this test covered it. Such code should be rolled back and reintroduced once the test is fixed.

If that isn't the case, and code coverage isn't a concern due to other tests covering the code, you can transform the test and introduce it as a green test.

Here, code is also being rolled back but just enough to cause the test to fail. If that's not enough to cover all the code introduced while only covered by the buggy test we need a bigger code roll back and more tests.

Introduce a green test

  • Run the test
    • Note the green bar
    • Break the code being tested
  • Run the test
    • Note the red bar
    • Un-break the code being tested
  • Run the test
    • Note the green bar

Breaking the code can be commenting out code or moving it elsewhere only to paste it back later. This shows us the scope of code the test covers.

For these last two runs you're right back into the normal red green cycle. You're just pasting instead of typing to un-break the code and make the test pass. So be sure you're pasting only enough to make the test pass.

The overall pattern here is to see the color of the test change the way we expect. Note that this creates a situation where you briefly have an un-trusted green test. Be careful about getting interrupted and forgetting where you are in these steps.

My thanks to RubberDuck for the Embracing the Red Bar link.

  • 2
    I like this answer best: It's important to see the test fail with incorrect code, so I would delete/comment out the code, correct the tests and see them fail, put back the code (maybe introduce a deliberate error to put the tests to the test) and correct to code to make it work. It's very XP to completely delete and rewrite it, but sometimes you just have to be pragmatic. ;) – GolezTrol Jan 29 '17 at 2:47
  • @GolezTrol I think my answer says the same thing, so I'd appreciate any feedback you had on whether that was unclear. – jonrsharpe Jan 29 '17 at 9:53
  • @jonrsharpe Your answer is good too, and I upvoted it before even reading this one. But where you are very strict in reverting the code, CandiedOrange suggests a more pragmatic approach that appeals more to me. – GolezTrol Jan 29 '17 at 13:19
  • @GolezTrol I didn't say how to revert the code; comment it out, cut and paste, stash it, use your IDE's history; it doesn't really matter. The crucial thing is why you do it: so you can check that you're getting the right failure. I've edited, hopefully to clarify. – jonrsharpe Jan 29 '17 at 13:21
10

What is the overall goal, you want to achieve?

  • Making nice tests?

  • Making the correct implementation?

  • Doing TTD religiously right?

  • None of the above?

Perhaps you overthink your relationship to tests and to testing.

Tests make no guarantees about the correctness of an implementation. Having all tests pass says nothing about, whether your software does what it should; it makes no essentialistic statements about your software.

Taking your example:

The "correct" implementation of the addition would be the code equivalent to a+b. And as long as your code does that, you would say the algorithm is correct in what it does and is correctly implemented.

int add(int a, int b) {
    return a + b;
}

On the first sight, we both would agree, that this is the implementation of an addition.

But what we are doing really is not saying, that this code is the implementation of addition it only behaves to a certain degree like one: think of integer overflow.

Integer overflow does happen in code, but not in the concept of addition. So: your code behaves to a certain extent like the concept of addition, but is not addition.

This rather philosophical point of view has several consequences.

And one is, that you could say, tests are nothing more than assumptions of expected behaviour of your code. In testing your code, you could (perhaps) never make sure, your implementation is right, the best you could say is, that your expectations on what results your code delivers were or weren't met; be it, that your code is wrong, be it, that your test is wrong or be it, that both of them are wrong.

Useful tests help you to fix your expectations on what the code should do: as long as I do not change my expectations and as long as the modified code gives me the result I am expecting, I could be sure, that the assumptions I made about the results seem to work out.

That doesn't help, when you made the wrong assumptions; but hey! at least it prevents schizophrenia: expecting different results when there should be none.


tl;dr

What is the best course of action in TDD if, after implementing the logic correctly, the test still fails (because there is a mistake in the test)?

Your tests are assumptions about the behaviour of the code. If you have good reason to think your implementation is right, fix the test and see if that assumption holds.

  • 1
    I think the question about overall goals is quite important, thanks for bringing it up. For me, the highest prio is the following: 1. correct implementation 2. "nice" tests (or, I would rather say, "useful"/"good designed" tests). I see TDD as a possible tool for achieving those two goals. So, while I don't want to necessarily religiously follow TDD, in the context of this question, I'm mostly interested in the TDD perspective. I'll edit the question to clarify this. – Attilio Jan 31 '17 at 18:23
  • So, would you write a test that tests for overflow and passes when it happens or would you make it fail when it happens because the algorithm is addition and overflow produces the wrong answer? – Jerry Jeremiah Feb 2 '17 at 1:24
  • 1
    @JerryJeremiah My point is: What your tests should cover is dependend on your use case. For a use case where you add up a bunch of single digits, the algorithm is good enough. If you know, that it is very likely that you add up "big numbers", the datatype is clearly the wrong choice. A test would reveal that: your expectation would be »works for big numbers« and is in several cases not met. Then the question would be how to deal with those cases. Are they corner cases? When yes, how to deal with them? Perhaps some quard clauses help to prevent a greater mess. The answer is context bound. – Thomas Junk Feb 2 '17 at 7:54
7

You need to know that the test is going to fail if the implementation is wrong, which isn't the same as passing if the implementation is right. Therefore you should put the code back into a state where you expect it to fail before correcting the test, and make sure it fails for the reason you expected (i.e. 5 != 12), rather than something else you didn't predict.

  • How can we check that the test is failing for the reason we expect? – Basilevs Jan 29 '17 at 9:03
  • 2
    @Basilevs you: 1. make a hypothesis as to what the reason for failure should be; 2. run the test; and 3. read the resulting failure message and compare. Sometimes this also suggests ways you could rewrite the test to give you a more meaningful error (for example, assertTrue(5 == add(2, 3)) gives a less useful output than assertEqual(5, add(2, 3)) even though they're both testing the same thing). – jonrsharpe Jan 29 '17 at 9:17
  • It is still unclear how to apply this principle here. I have a hypothesis - test returns a constant value, how running same test again would ensure I am right? Obviously to test that, I need ANOTHER test. I suggest to add explicit example to answer. – Basilevs Jan 29 '17 at 9:37
  • 1
    @Basilevs what? Your hypothesis here at step 3 would be "the test fails because 5 is not equal to 12". Running the test will show you whether the test fails for that reason, in which case you proceed, or for some other reason, in which case you figure out why. Perhaps this is a language problem, but it's unclear to me what you're suggesting. – jonrsharpe Jan 29 '17 at 9:41
5

In this particular case, if you change the 12 to an 11, and the test now passes, I think you've done a good job of testing the test as well as the implementation, so there's not much need to go through additional hoops.

However, the same issue can come up in more complex situations, such as when you have a mistake in your setup code. In that case, after fixing your test, you should probably try mutating your implementation in such a way to get that particular test to fail, and then revert the mutation. If reverting the implementation is the easiest way to do that, then that is fine. In your example, you might mutate a + b to a + a or a * b.

Alternatively, if you can mutate the assertion slightly and see the test fail, that can be pretty effective at testing the test.

0

I'd say, this is a case for your favorite version control system:

  1. Stage the correction of the test, keeping your code changes in your working directory.
    Commit with a corresponding message Fixed test ... to expect correct output.

    With git, this might require use of git add -p if test and implementation are in the same file, otherwise you can obviously just stage the two files separately.

  2. Commit the implementation code.

  3. Go back in time to test the commit made in step 1, making sure that the test actually fails.

You see, that way you do not rely on your editing prowess to move your implementation code out of the way while you test your failing test. You employ your VCS to save your work and to ensure that the VCS recorded history correctly includes both the failing and the passing test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.