How to correct a mistake in the test, after writing implementation

Question

What is the best course of action in TDD if, after implementing the logic correctly, the test still fails (because there is a mistake in the test)?

For example, suppose you would like to develop the following function:

int add(int a, int b) {
    return a + b;
}

Suppose we develop it in the following steps:

1. Write test (no function yet):

   // test1
   Assert.assertEquals(5, add(2, 3));
   

   Results in compilation error.

2. Write a dummy function implementation:

   int add(int a, int b) {
       return 5;
   }
   

   Result: test1 passes.

3. Add another test case:

   // test2 -- notice the wrong expected value (should be 11)!
   Assert.assertEquals(12, add(5, 6));
   

   Result: test2 fails, test1 still passes.

4. Write real implementation:

   int add(int a, int b) {
       return a + b;
   }
   

   Result: test1 still passes, test2 still fails (since 11 != 12).

In this particular case: would it be better to:

1. correct test2, and see that it now passes, or
2. delete the new portion of implementation (i.e. go back to step #2 above), correct test2 and let it fail, and then reintroduce the correct implementation (step #4. above).

Or is there some other, cleverer way?

While I understand that the example problem is rather trivial, I'm interested in what to do in the generic case, which might be more complex than the addition of two numbers.

EDIT (In response to the answer of @Thomas Junk):

The focus of this question is what TDD suggests in such a case, not what is "the universal best practice" for achieving good code or tests (which might be different than the TDD-way).

Answer 1

The absolutely critical thing is that you see the test both pass and fail.

Whether you delete the code to make the test fail then rewrite the code or sneak it off to the clipboard only to paste it back later doesn't matter. TDD never said you had to retype anything. It wants to know the test passes only when it should pass and fails only when it should fail.

Seeing the test both pass and fail is how you test the test. Never trust a test you've never seen do both.

---

Refactoring Against The Red Bar gives us formal steps for refactoring a working test:

• Run the test
  • Note the green bar
  • Break the code being tested
• Run the test
  • Note the red bar
  • Refactor the test
• Run the test
  • Note the red bar
  • Un-break the code being tested
• Run the test
  • Note the green bar

However, we aren't refactoring a working test. We have to transform a buggy test. One concern is code that was introduced while only this test covered it. Such code should be rolled back and reintroduced once the test is fixed.

If that isn't the case, and code coverage isn't a concern due to other tests covering the code, you can transform the test and introduce it as a green test.

Here, code is also being rolled back but just enough to cause the test to fail. If that's not enough to cover all the code introduced while only covered by the buggy test we need a bigger code roll back and more tests.

Introduce a green test

• Run the test
  • Note the green bar
  • Break the code being tested
• Run the test
  • Note the red bar
  • Un-break the code being tested
• Run the test
  • Note the green bar

Breaking the code can be commenting out code or moving it elsewhere only to paste it back later. This shows us the scope of code the test covers.

For these last two runs you're right back into the normal red green cycle. You're just pasting instead of typing to un-break the code and make the test pass. So be sure you're pasting only enough to make the test pass.

The overall pattern here is to see the color of the test change the way we expect. Note that this creates a situation where you briefly have an un-trusted green test. Be careful about getting interrupted and forgetting where you are in these steps.

My thanks to RubberDuck for the Embracing the Red Bar link.

Answer 2

What is the overall goal, you want to achieve?

• Making nice tests?

• Making the correct implementation?

• Doing TTD religiously right?

• None of the above?

Perhaps you overthink your relationship to tests and to testing.

Tests make no guarantees about the correctness of an implementation. Having all tests pass says nothing about, whether your software does what it should; it makes no essentialistic statements about your software.

Taking your example:

The "correct" implementation of the addition would be the code equivalent to a+b. And as long as your code does that, you would say the algorithm is correct in what it does and is correctly implemented.

int add(int a, int b) {
    return a + b;
}

On the first sight, we both would agree, that this is the implementation of an addition.

But what we are doing really is not saying, that this code is the implementation of addition it only behaves to a certain degree like one: think of integer overflow.

Integer overflow does happen in code, but not in the concept of addition. So: your code behaves to a certain extent like the concept of addition, but is not addition.

This rather philosophical point of view has several consequences.

And one is, that you could say, tests are nothing more than assumptions of expected behaviour of your code. In testing your code, you could (perhaps) never make sure, your implementation is right, the best you could say is, that your expectations on what results your code delivers were or weren't met; be it, that your code is wrong, be it, that your test is wrong or be it, that both of them are wrong.

Useful tests help you to fix your expectations on what the code should do: as long as I do not change my expectations and as long as the modified code gives me the result I am expecting, I could be sure, that the assumptions I made about the results seem to work out.

That doesn't help, when you made the wrong assumptions; but hey! at least it prevents schizophrenia: expecting different results when there should be none.

---

tl;dr

What is the best course of action in TDD if, after implementing the logic correctly, the test still fails (because there is a mistake in the test)?

Your tests are assumptions about the behaviour of the code. If you have good reason to think your implementation is right, fix the test and see if that assumption holds.

Answer 3

You need to know that the test is going to fail if the implementation is wrong, which isn't the same as passing if the implementation is right. Therefore you should put the code back into a state where you expect it to fail before correcting the test, and make sure it fails for the reason you expected (i.e. 5 != 12), rather than something else you didn't predict.

Answer 4

In this particular case, if you change the 12 to an 11, and the test now passes, I think you've done a good job of testing the test as well as the implementation, so there's not much need to go through additional hoops.

However, the same issue can come up in more complex situations, such as when you have a mistake in your setup code. In that case, after fixing your test, you should probably try mutating your implementation in such a way to get that particular test to fail, and then revert the mutation. If reverting the implementation is the easiest way to do that, then that is fine. In your example, you might mutate a + b to a + a or a * b.

Alternatively, if you can mutate the assertion slightly and see the test fail, that can be pretty effective at testing the test.

Answer 5

I'd say, this is a case for your favorite version control system:

1. Stage the correction of the test, keeping your code changes in your working directory.
   Commit with a corresponding message Fixed test ... to expect correct output.

   With git, this might require use of git add -p if test and implementation are in the same file, otherwise you can obviously just stage the two files separately.

2. Commit the implementation code.

3. Go back in time to test the commit made in step 1, making sure that the test actually fails.

You see, that way you do not rely on your editing prowess to move your implementation code out of the way while you test your failing test. You employ your VCS to save your work and to ensure that the VCS recorded history correctly includes both the failing and the passing test.