0

I'm doing TDD. Let's say I expect some method m of some class C to call some function f (to keep things really simple).

How would I design a unit test which would pick-up and therefore fail, if m were to be implemented something like:

m(func f)
{
  // let's say some_internal_counter starts at 0 when object initialised.
  if (some_internal_counter < 10)
  {
    f();
    some_internal_counter++;
  }
}

A unit test like the following would not only pass but also result in 100% code-coverage:

C c = new C(); // C.some_internal_counter at 0, but the one writing this test doesn't know (or shouldn't know) this.
func f = SomeMockFunc();
c.m(f);
Expect(f).ToBeCalled();

But it hasn't safeguarded against being dependent on some (unauthorised) internal state. So the program would make its way to prod (since everything's green), and not work when m is called more than 10 times.

  • Based on answers, I realise I should've mentioned that (formally) I'm doing black-box testing. – Ash Apr 10 '18 at 0:31
  • If somebody could write this kind of implementation - nothing will stop him from changing tests as well to psss – Fabio Apr 10 '18 at 4:25
  • @Fabio Not if they don't have access to the test suite. – Ash Apr 10 '18 at 4:29
  • seems strange that developer who writing code do not have access to the tests ;). For this kind of defensive - good tool is code review. Where you can review code written by developers you don't trust before committing it to the master branch. – Fabio Apr 10 '18 at 6:14
  • 1
    Someone once said that C++ was designed to protect against Murphy not Machiavelli. The same can be said of unit tests. It will always be possible for a malevolent developer to design an implementation, like your example, that passes all the unit tests yet is still incorrect in other cases. Good test coverage means this should be unlikely to happen by accident, and not hiring maniacs should prevent all other cases. – Sean Burton Apr 23 '18 at 16:09
7

Edsger W. Dijkstra:

Testing shows the presence, not the absence of bugs.

If you want to prove the absence of bugs, you need to turn to formal verification rather than tests. But as Donald Knuth said:

Beware of bugs in the above code; I have only proved it correct, not tried it.

More seriously, TDD is never ever going to protect you against actively malicious or misguided programmers. If you're doing red-green-refactor TDD then it should be clear from a quick review that this code is not the simplest thing that could make the test pass. From that point you can figure out what the additional requirement is (if it exists) and implement a corresponding test. Finding unnecessary (and therefore often buggy) code can be a very powerful side effect of doing TDD.

  • Ah yes, I should've remembered that ol' statement by Dijkstra. I was hoping there was some clever design pattern that could be implemented in your unit test which would make looking for such bugs unnecessary during the Refactor stage of TDD. We talk design patterns for application code all the time but rarely do we hear about them when it comes to unit tests. – Ash Apr 10 '18 at 0:17
  • @Ash: There are automated tools that will write tests specifically to exercise boundary conditions. See google.com/search?q=automated+white+box+testing – Robert Harvey Apr 10 '18 at 0:18
  • @RobertHarvey Unfortunately I'm doing black box testing : - ) – Ash Apr 10 '18 at 0:27
  • @Ash: Um, how are you going to test the internal state of an object (a white box test, by definition), if you are limited to black box testing? – Robert Harvey Apr 10 '18 at 0:40
  • @RobertHarvey Precisely what I would like to know. I'm asking for the impossible it seems... – Ash Apr 10 '18 at 0:42
1

Unit testing is generally white box testing. You write and read the code and tests together and you think about their quality together. You can never write a test that would stop sufficiently pathological code passing and giving users a bad time.

If you're doing tdd as described by Bob Martin you never would have written the function as shown. Martin advocates only writing the minimum code necessary to pass the tests. The implementation of m is much more complex than required to pass the tests.

On the other hand if you're not doing tdd but you're writing the unit test after the code you should look at the code and make an effort to test the various ways it can run. Whenever you see an if condition in the code you would ideally write one test for the false branch and one for the true branch.

  • 2
    Bob Martin-style unit tests are black box, by definition. – Robert Harvey Apr 10 '18 at 0:19
  • @bdsl I agree with Robert regarding Bob Martin being black box. The issue of white box testing and writing the unit test after the code is irrelevant here. The point about writing the minimum amount of code however is valid, however, I was hoping for a way to avoid not relying on the Refactor stage of TDD to identify this. Could you please edit your answer. – Ash Apr 10 '18 at 0:24
  • @RobertHarvey, if you don't know what the code currently does and doesn't do, how do you write a failing test other than by sheer luck? In order to write a failing test, you have to inspect the code, therefore it is white box testing. – David Arno Apr 10 '18 at 13:57
  • @DavidArno: No. You write the test to confirm a specific output from your unit, given a specific input. Your test never has knowledge of the unit's internals. Naturally, you have knowledge of the code, but the test does not. – Robert Harvey Apr 10 '18 at 15:22
  • 1
    @DavidArno: [shrug] The point is that TDD doesn't test internals directly, but only indirectly by probing the unit's external API. – Robert Harvey Apr 10 '18 at 15:44
-2

safeguarded against being dependent on some (unauthorised) internal state

Unit tests aren't for that. Use your brain, eyes and fingers, and remove the variables that track state.

  • 1
    By that logic, we don't even need unit tests...just use your brain, eyes and fingers. – Ash Apr 22 '18 at 1:12
  • @Ash No, that's doesn't follow. Unit tests are not for what was suggested. You should trying to contribute something intelligent here instead of your straw man attacks. – Josh May 1 '18 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.