-1

I'm doing TDD. Let's say I expect some method m of some class C to call some function f (to keep things really simple).

How would I design a unit test which would pick-up and therefore fail, if m were to be implemented something like:

m(func f)
{
  // let's say some_internal_counter starts at 0 when object initialised.
  if (some_internal_counter < 10)
  {
    f();
    some_internal_counter++;
  }
}

A unit test like the following would not only pass but also result in 100% code-coverage:

C c = new C(); // C.some_internal_counter at 0, but the one writing this test doesn't know (or shouldn't know) this.
func f = SomeMockFunc();
c.m(f);
Expect(f).ToBeCalled();

But it hasn't safeguarded against being dependent on some (unauthorised) internal state. So the program would make its way to prod (since everything's green), and not work when m is called more than 10 times.

13
  • Based on answers, I realise I should've mentioned that (formally) I'm doing black-box testing.
    – Ash
    Commented Apr 10, 2018 at 0:31
  • If somebody could write this kind of implementation - nothing will stop him from changing tests as well to psss
    – Fabio
    Commented Apr 10, 2018 at 4:25
  • @Fabio Not if they don't have access to the test suite.
    – Ash
    Commented Apr 10, 2018 at 4:29
  • seems strange that developer who writing code do not have access to the tests ;). For this kind of defensive - good tool is code review. Where you can review code written by developers you don't trust before committing it to the master branch.
    – Fabio
    Commented Apr 10, 2018 at 6:14
  • 1
    Someone once said that C++ was designed to protect against Murphy not Machiavelli. The same can be said of unit tests. It will always be possible for a malevolent developer to design an implementation, like your example, that passes all the unit tests yet is still incorrect in other cases. Good test coverage means this should be unlikely to happen by accident, and not hiring maniacs should prevent all other cases. Commented Apr 23, 2018 at 16:09

3 Answers 3

7

Edsger W. Dijkstra:

Testing shows the presence, not the absence of bugs.

If you want to prove the absence of bugs, you need to turn to formal verification rather than tests. But as Donald Knuth said:

Beware of bugs in the above code; I have only proved it correct, not tried it.

More seriously, TDD is never ever going to protect you against actively malicious or misguided programmers. If you're doing red-green-refactor TDD then it should be clear from a quick review that this code is not the simplest thing that could make the test pass. From that point you can figure out what the additional requirement is (if it exists) and implement a corresponding test. Finding unnecessary (and therefore often buggy) code can be a very powerful side effect of doing TDD.

6
  • Ah yes, I should've remembered that ol' statement by Dijkstra. I was hoping there was some clever design pattern that could be implemented in your unit test which would make looking for such bugs unnecessary during the Refactor stage of TDD. We talk design patterns for application code all the time but rarely do we hear about them when it comes to unit tests.
    – Ash
    Commented Apr 10, 2018 at 0:17
  • @Ash: There are automated tools that will write tests specifically to exercise boundary conditions. See google.com/search?q=automated+white+box+testing Commented Apr 10, 2018 at 0:18
  • @RobertHarvey Unfortunately I'm doing black box testing : - )
    – Ash
    Commented Apr 10, 2018 at 0:27
  • @Ash: Um, how are you going to test the internal state of an object (a white box test, by definition), if you are limited to black box testing? Commented Apr 10, 2018 at 0:40
  • @RobertHarvey Precisely what I would like to know. I'm asking for the impossible it seems...
    – Ash
    Commented Apr 10, 2018 at 0:42
1

Unit testing is generally white box testing. You write and read the code and tests together and you think about their quality together. You can never write a test that would stop sufficiently pathological code passing and giving users a bad time.

If you're doing tdd as described by Bob Martin you never would have written the function as shown. Martin advocates only writing the minimum code necessary to pass the tests. The implementation of m is much more complex than required to pass the tests.

On the other hand if you're not doing tdd but you're writing the unit test after the code you should look at the code and make an effort to test the various ways it can run. Whenever you see an if condition in the code you would ideally write one test for the false branch and one for the true branch.

6
  • 2
    Bob Martin-style unit tests are black box, by definition. Commented Apr 10, 2018 at 0:19
  • @bdsl I agree with Robert regarding Bob Martin being black box. The issue of white box testing and writing the unit test after the code is irrelevant here. The point about writing the minimum amount of code however is valid, however, I was hoping for a way to avoid not relying on the Refactor stage of TDD to identify this. Could you please edit your answer.
    – Ash
    Commented Apr 10, 2018 at 0:24
  • @RobertHarvey, if you don't know what the code currently does and doesn't do, how do you write a failing test other than by sheer luck? In order to write a failing test, you have to inspect the code, therefore it is white box testing.
    – David Arno
    Commented Apr 10, 2018 at 13:57
  • @DavidArno: No. You write the test to confirm a specific output from your unit, given a specific input. Your test never has knowledge of the unit's internals. Naturally, you have knowledge of the code, but the test does not. Commented Apr 10, 2018 at 15:22
  • 1
    @DavidArno: [shrug] The point is that TDD doesn't test internals directly, but only indirectly by probing the unit's external API. Commented Apr 10, 2018 at 15:44
-2

safeguarded against being dependent on some (unauthorised) internal state

Unit tests aren't for that. Use your brain, eyes and fingers, and remove the variables that track state.

2
  • 1
    By that logic, we don't even need unit tests...just use your brain, eyes and fingers.
    – Ash
    Commented Apr 22, 2018 at 1:12
  • @Ash No, that's doesn't follow. Unit tests are not for what was suggested. You should trying to contribute something intelligent here instead of your straw man attacks.
    – Josh
    Commented May 1, 2018 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.