What is black box unit testing?

Question

I recently had my final exam for a software engineering course for my masters program and one of the questions on the exam was the following:

Unit Testing is considered:
a. White-box Testing
b. Black-box Testing
c. Either

In my 7 years of software development experience, unit testing has always taken a white box approach. The tester has always had full knowledge of the implementation of the unit while writing the tests. Black box testing always came later in the forms of integration, system, and acceptance testing.

However, the correct answer to the exam (according to the professor) is that unit testing can be either white or black box testing.

I have done some research, and it seems many cases "black box unit testing" is used to describe a test-first approach where the unit tests are written before the code is. However in my opinion this is still white box testing. While the implementation does not yet exist, whoever is writing the test generally has a pretty good idea about how the source code is going to be implemented.

Can someone please explain to me how black box unit testing works (if it truly is a thing) and how it differs from white box unit testing?

Answer 1

Your professor is right: unit testing can be either black-box or white-box. The difference is less about what the tester knows, but more about how you generate test cases.

With black box testing, you only look at the interface and (if it exists) the specification for a component. When a function has a signature int foo(int a, int b), then I can immediately generate a few test cases just by testing interesting integers: zero, one, minus one, multi-digit numbers, INT_MAX, INT_MAX - 1 and so on. Black box tests are great because they are independent from the implementation. But they might also miss important cases.

With a white-box test, I look at the implementation, i.e. the source code and generate test cases from that. For example, I might want to achieve 100% path coverage for a function. I then choose input values so that all paths are taken. White-box tests are great because they can exhaustively exercise a piece of code, with far more confidence than a black-box tests. But they might be only testing implementation details, not actually important behaviour. In some cases, they are clearly a waste of time.

Since a white-box test is derived from the implementation, it can only be written afterwards. A black-box test is derived from the design/interface/specification, and can therefore be written before or after the implementation. TDD is neither clearly black-box or white-box. Since all behaviour is first expressed by a test and then the minimal code for that behaviour is implemented, TDD results in similar test cases to a white box test. But when we look at the information flow, TDD tests are not derived from the source code, but from external requirements. Therefore, TDD is more black-box-like.

Answer 2

If you're undertaking Test-Driven Development, then in theory all your unit-testing should be black-box. This is your "test-first approach". You write the contract (interface), write the tests for that contract, and then the contract is fulfilled by the implementation. The test therefore knows nothing, and should know nothing, about the implementation.

After all, when you write a test, what are you testing? Public methods/functions.

If you were to write the interface for a class, and then write the tests, and then you get hit by a bus, the guy who writes the class while you're in hospital should be able to do so from your interface, right? He shouldn't have to throw it away and write his own interface and tests.

Where this falls apart somewhat is when you need to mock something that the implementation depends on, but if you find yourself in the situation whereby you're mocking something that is never exposed publically, then you've made a mistake, and you need to look to Dependency Injection et al. Therefore I would argue that white-box unit-testing, not black, ought to be the exception.

Consider 'Testing on the Toilet - Test Behaviour Not Implementation', wherein a class's implementation is altered but the tests should still be valid.

However, if you need to make sure your code coverage is up (i.e. make sure all the conditional paths are tested within the implementation), then you would absolutely need to white-box unit-test, because the only way you can know what your paths are is by looking at the paths in the implementation.

Answer 3

I would argue that all well-written unit tests are inherently "black box". Sure I might have an implementation in mind when I write the test, but that implementation may change when I refactor. So the test should only use public APIs during the test in order to test the functionality, not the implementation. It doesn't care about the implementation details, so its black box testing.

If I write tests that access internal or private aspects of the unit under test, then I'm testing the implementation details: I'm white box testing. But I'm also writing brittle tests that can easy break when the implementation is changed. So such white box tests are a bad idea and should be avoided.

Conclusion: if you white box test with unit tests, you have poorly constructed tests. Only back box test with those unit tests. You professor is correct: it can be either. But only if done badly.

Answer 4

I was just in the process of writing unit tests which perform black-box testing. That is, I am testing public methods in a class and by implication of the results testing logic in the private methods they call.

I do this by changing inputs to the public method being unit tested and testing expected outputs that are determined or mutated by logic in the supporting private methods, the implementation of which, my "unit tests" need know nothing about.

So, there is nothing stopping you performing black box testing on unit tests and the tests will break if someone messes with the implementation of the hidden supporting logic. In fact, this seems like a superior, more efficient, approach than white box unit testing everything in a class for the sake of it. I'm with the professor.