Alternatives for Unit testing

Question

The problem I have with unit testing is that while it makes refactoring(regression testing) easier, it expands the code base and make it more difficult to prototype or to change the design. Developers tend to be averse to change the API due to unit testing.

Is it possible to develop a testing system such that you don't have to write tests?

For an example, it would show the resultant values for every value in the domain ( a model of the function) and the programmer can choose to accept it as the test. That way you can diff the model to find regression bugs unlike property based testing.

Answer 1

This is impossible:

• Unless you only restrict yourself to pure functions, the result value may not capture everything a subroutine does. Many subroutines have useful side-effects; these side-effects are not captured in the result value. E.g., C's printf returns the number of characters written, but the more interesting feature of printf is that it writes something, and that is not captured in the return value.
• It is infeasible for large domains to show the result for every value in the domain, e.g. a Java method foo(long, long, long) has 2¹⁹² (~10⁶⁴) different possible inputs, that's close to the number of particles in the universe, and even if you had a 1000000 core CPU with 10GHz that could show one result for every clock cycle, it would take you ~20000000000000000000000000000000000 years. Even worse: most domains are infinite, e.g. there are infinitely many Strings, infinitely many Ruby Integers, etc.
• It is impossible to determine what the result of a subroutine is without running it. (This is a variation of the Function Problem which is provably undecidable.) Heck, it is even impossible to determine whether it even has a result or not. (This is called the Halting Problem and is provably undecidable.) So, the only way to generate the result values is to run the subroutine for every combination of possible inputs, but you have no idea what the subroutine does! It might format your hard disk, for example. (And you can't figure out what it does, because of the Function Problem.)

Answer 2

It depends on your application. If you write a converter and you have a series of fixed input sets you could convert those and report any differences with a prior conversion output. This is a valuable test. If your work is for one client at a time (your project is for one set of input data), that could work.

But then, although an alternative, it would not provide the same information as unit tests. It would just be a limited regression test.

• code coverage would be unknown;
• the test cycle would likely be longer, doing a full run on your test data will be time consuming and also make developers reluctant to do it after each change;
• the reference output may have errors from the beginning, you will never be sure.

In the end it is a matter of perception. Do you still see making tests as a cumbersome after-overhead task that the environment expects from you but you would rather skip so you can move ahead, or do you accept and embrace writing fitting tests as part of the development work and of your personal definition of done?

Apparently your developers feel the pressure to cut some corners. Or writing tests is just boring work to them. Perhaps they do not get "points" for them.

Is there a need to be strict on writing unit tests? Do you see a lot of bugs coming back that would have been caught by proper unit tests?

To some applications unit tests are invaluable. To others they may be no more than a pacifier. If you have clear fine-grained requirements you can hardly go wrong with them. If you only have one big requirement and you have to figure out all the small steps yourself, unit tests will not do much for you. They will merely affirm that your little trial does what you had in mind, not if it brings you any closer to your goal.

So as always, it depends. Think hard about what matters to your project, what makes sense to test and test that. The bugs you see may be an indicator.

Answer 3

It is possible to create correct programs without writing tests, but you would not use a testing system. You would use an automatic proof system, combined with a specification of behavior. The best ones work directly with the compiler so that your specification is part of your code, in the style originally promoted by Liskov.

In real life, you will need to write tests. You can avoid many low level tests by including precondition, postcondition, and invariant checks in all low-level run-time code. This way, any test will include testing of low level code in a fairly thorough manner. All tests are simply a poor substitute for the real goal, which is a correctness proof.