7

I have been taught that with TDD, the tests "naturally fail first but it is a good habit to run them anyway to see the red light". Well, but I am quite sure a unit test written first for a new feature might actually pass, if such a feature is already implemented (and for example somehow inhibited).

I am borrowing the following example from this answer:

    public string ShowScoreEvaluation(byte points)
    {
        switch(points)
         case 3:
            return "You are good!";
         case 2:
            return "Not bad!";
         case 1:
            return "Quite bad";
         case 0:
            return "You suck!"
      return null;

    }

    //caller code
    if (Points>0)
      Display(ShowScoreEvaluation(Points));

In the code above, the calling code does not expect to call the method when Points=0. Maybe during the implementation of that method, the programmer just put something there (as a joke or a placeholder) even for the case when points=0.

And now imagine, that you join the project and get a new request "When player has 0 points, show an encouraging message blabla". You write a unit test with Points=0 and expecting a string with length>0...and it did not fail, although you would expect it.

Could this not happen in the real life? I mean, to me this is the reason why I truly should see whether a new test fails, because then how could I know what made it pass?

  • The test shouldn't expect a string with length > 0. It should expect a string with encouragement. In which case "fail first" is verified. – mouviciel Sep 11 '18 at 18:41
16

I guess for what you really want to know, your example is not well suited, so let me describe a better one.

There are several programming problems where the most elegant, shortest implementation is not the one which just serves a few test cases, but one which solves the problem in a more general fashion than specified by the tests.

For example, one can implement a sorting algorithm incrementally by writing some test cases in a TDD fashion for sorting two, three or four elements. Maybe one starts with an implementation which can only sort two elements, like

int[] Sort(int[] list)
{
     // ... maybe some handling for lists with 0 or 1 element here ...
     if(list[0]>list[1])
        return new[]{list[1],list[0]};
     else
        return new[]{list[0],list[1]};
}

Now, a test which passes 3 elements into this function will fail. However, when you extend this code to 3, 4 or more elements, you will quickly reach the point where a general algorithm like a bubble sort or insertion sort (which works for arbitrary list lengths) is simpler than an implementation which works only for a fixed number of elements. In fact, by doing TDD, you might have started with a convoluted algorithm which compared up to 4 elements one-by-one against each other, but in the refactoring step, you replaced this by a more general sorting algorithm, which makes the overall implementation simpler.

Now it should not be very astonishing when you add further tests with more elements, these tests won't fail, though you followed all the TDD rules literally, and though you did not implement more code than needed to satisfy all existing tests.

And yes, that is a real-world case, far more likely to happen as this answer pretends. I have encountered this situation lots of times for all different kind of problems in string processing, set manipulation, mathematical or geometrical algorithms: general implementations are very often simpler than specialized ones.

So what should one do in this case? Leaving out the additional tests with 5, 8 or 20 elements, just because the code "is already complete, and the additional tests won't induce any more code changes"? I would not recommend this - it is obviously good to have such additional tests for a complex algorithm, it will give you much more confidence into the correctness of the code.

The better alternative here is to make your additional tests artificially fail for a short time. The main purpose of seeing any tests failing first before they pass in TDD is to make sure the test is actually executed - it is a "test for the test". You could, for example, add a statement like

if(list.Length>=5)
    return null;

somewhere inside the Sort function, and remove it after you saw it the test failing. That proofs your new tests are actually executed, and you did not intermix it with some existing test.


Since you asked for another example: lets say you have a function

`string TrimNumeric(string value)`

which is expected to replace non-numeric characters from the beginning or end of the input string value and return the result. What happens with non-numeric characters in the middle is not specified so far. The following test cases are already passing:

Assert.AreEqual("123",TrimNumeric("abc123"));
Assert.AreEqual("123",TrimNumeric("123xyz"));
Assert.AreEqual("456",TrimNumeric("abc456xyz"));

Now you get an additional requirement: non-numeric characters in the middle shall be stripped as well. You start by adding a test:

Assert.AreEqual("123",TrimNumeric("1a2b3cxyz"));

In such a case, if you don't know how TrimNumeric is implemented internally, there is no indication if this test will fail or not. Indeed, if TrimNumeric was implemented in a straightforward, simple, and general fashion, by iterating over all characters and keeping only the numeric ones, it is IMHO pretty likely this test will pass immediately. Nevertheless it should be clear why is necessary to write such a test. Maybe the test will not pass, if the implementation looks differently. But if it passes right from the start, make sure you make it fail at least temporarily, to be sure it gets executed.

  • Thank you, that was really helpful. Could I ask, would it be possible to show another situation where you get a new request, write a unit test and it passes, although you would expect it to fail? I am unsure about the one in the accepted answer here: stackoverflow.com/questions/49357004/… – John V May 19 '18 at 10:25
  • @user970696: I think that SO question & answer is pretty mediocre, the example in the accepted answer seems seems to have the same weakness as the one you from the other answer you picked for this question. But see my edit. – Doc Brown May 19 '18 at 11:17
  • 4
    A better example (still in sorting) is that your current implementation just happens to be stable, and then you get a request specifically for that. There are a vast number of invariants that code can have, without any intent that it must – Caleth May 19 '18 at 11:48
  • This is a great answer to the general issue. I think where our other answer got sidetracked was focusing on the original questions emphasis of "expecting the test to fail". In situations like you describe here, I agree that adding the extra test cases is very good for capturing the intent of the code. But in these situations I would not be "expecting" the new test cases to fail when I write it. I would actually be expecting them to pass. – Caleb May 19 '18 at 17:44
  • 1
    @Caleb: thanks for your comment (I also upvoted your answer before I wrote mine). About the failure expectation: the idea behind my TrimNumeric example was to show a case where it is not inherently clear what to expect. – Doc Brown May 19 '18 at 20:29
3

I am quite sure a unit test written first for a new feature might actually pass

Yes, it might. There are a number of reasons why a feature might already 'be there', untested, but if you are following strict TDD, the most common reason would be that the refactor step in your red-green-refactor cycle made the behaviour of your code more general, and able to cope with cases beyond your initial specific test cases.

The other most common reason in a professional context is probably that that someone has previously worked on the code and put the feature in there without a matching test!

In either of those cases, it's still well-worth writing the test. You should only consider that your product 'has' a feature if that feature is tested and documented.

1

Your philosophy is correct.

(but your example is not a valid argument.)

The idea of a TDD test failing "naturally" is not really part of the rules. And you are adding an assumption that code written while not following the rules, will still obey your assumptions.

The rules of TDD state:

  1. You are not allowed to write any production code unless it is to make a failing unit test pass.
  2. You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
  3. You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

(emphasis mine)

So when you say this:

... a unit test written first for a new feature might actually pass, if such a feature is already implemented...

That code should have already had tests written for it. And when writing those tests, they should have "failed" before the code was written.

So why do I say your philosophy is correct?

It all depends on what test case you choose to write first.

Let's say that I am going to be create a function that prints out every character in a string passed into it. My goal is to end up with something like this (pseudo code):

function printLetters (String input) {
    for(var letter in input) {
        console.print(letter);
    }
}

Now, if I am sitting down to work in this function, my first test case might be this:

"When an empty string is passed into my function, nothing is printed".

Now, by rule 2, if we count "compile errors" as failing test, then once I try to call the function (that I haven't created yet) I have a failing test! YAY!

But, what WILL happen is that eventually I would start writing the prod code, and get to this point:

function printLetters (String input) {

}

And now my test is passing. So, I know that my code is not done, but I don't have a failing test.

But what does this mean?

This just means that we have not written the rest of the test cases to satisfy the business rules!

So I need to add more test cases (which will fail).

And, if I am following the "rules", I better make sure that the test I am writing is failing (AND FAILING FOR THE CORRECT REASON!) before I start writing the production code.

  • Hmm thanks, so you think in practice it cannot happen? That a unit test expected to fail is actually passing? I remember an article where they claim that if this happens, it means the assumptions about the production code are wrong and common reason is that such a thing is already implemented. – John V May 19 '18 at 8:06
  • What would you think about the accepted answer here? stackoverflow.com/questions/49357004/… – John V May 19 '18 at 8:06
  • "in practice" can be interpreted 2 ways here. Assuming that the entire code base is covered (correctly) with tests / assuming that some of the code is not tested completely. In reality, the second is always what is true. (that is why "bugs" exist in software). – Caleb May 19 '18 at 8:18
  • 1
    Just because a test passes does not mean that the test is "correct". "A == A" will pass the test. but it probably not what I was really TRYING to validate in my test. – Caleb May 19 '18 at 8:22
  • 1
    "If you add a unit test and it passes automatically, it simply means that the code is ALREADY doing what you wanted it to do" - Or it means that the test isn't actually testing what you intended it to test at all. – Sean Burton May 21 '18 at 11:26
0

Could this not happen in the real life?

In an infinite universe, almost anything can happen... but if the desired behaviour already exists, it is very unlikely (not impossible) that you would receive a feature request for it.

...[you] get a new request "When player has 0 points, show an encouraging message blabla". You write a unit test with Points=0 and expecting a string with length>0...

The problem is that your suggested test does not match the feature you are trying to drive out.

The test implied by your feature would be “with Points=0, expect a string matching "an encouraging message blabla"” -- which would fail for the current implementation.

Once you have seen it fail, you would make the simplest change possible to make it pass. What would that be? It depends on what other tests are in place.

I would guess the simplest change would be:

    switch(points)
     case 3:
        return "You are good!";
     case 2:
        return "Not bad!";
     case 1:
        return "Quite bad";
     case 0:
        return "an encouraging message blabla"
  return null;

But if you only have existing tests for cases 1-3, the simplest change could be:

    switch(points)
     case 3:
        return "You are good!";
     case 2:
        return "Not bad!";
     case 1:
        return "Quite bad";
  return "an encouraging message blabla";

This, in turn, could lead to a further requirement to return something else in a default case.

  • You are missing the point. The feature could be implemented simply because I programmed puts there a placeholder, in simple cases like that. The calling code, however, cannot use this implementation because it follows the specifications which say 1-3. – John V May 19 '18 at 8:03
  • 3
    If you are coding TDD, you would not have used a placeholder without adding a test which required it. – Chris Throup May 19 '18 at 8:05
  • Sure, but that requires discipline / diligence. So I believe this might happen as any other bug - by accident/oversight? – John V May 19 '18 at 8:08
  • 2
    TDD is a discipline. – Chris Throup May 19 '18 at 8:09
  • If TDD is not being followed, this could happen. So if you are trying to start TDD on an existing codebase, you may need to force the test to fail first (eg test for the wrong value, then test for the correct value). The point of failing first is to prove the test actually covers the behaviour it is supposed to. – Chris Throup May 19 '18 at 8:12
0

Could this not happen in the real life?

It does to me; especially when I happen to choose a simple initial test, where the default implementation happens to return the correct value.

For instance, suppose we are trying to develop a function that adds up the numbers in a list. If we happen to choose an empty list as our first test (why not? it's easy to set up), then the implementation return 0; is going to produce the correct result.

I have been taught that with TDD, the tests "naturally fail first but it is a good habit to run them anyway to see the red light"

What I eventually came to decide is that RED-GREEN is about test calibration; the goal during that part of the process is to verify that the test you have created actually evaluates your production implementation.

RED/GREEN is the most common pattern of calibration activities, but sometimes GREEN/RED/GREEN (or, if you prefer, GREEN/RED/REVERT) happen instead.

Calibrated tests that pass naively are still valuable, in that they constrain your choices during the refactoring phase (more precisely, they help to distinguish changes that are refactorings from changes that have altered the observable behavior of your system).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.