Are asserts or unit tests more important?

Question

Both asserts and unit tests serve as documentation for a codebase, and a means of discovering bugs. The main differences are that asserts function as sanity checks and see real inputs, whereas unit tests run on specific simulated inputs and are tests against a single well-defined "right answer". What are the relative merits of using asserts vs. unit tests as the main means of verifying correctness? Which do you believe should be emphasized more heavily?

Answer 1

Asserts are useful for telling you about the internal state of the program. For example, that your data structures have a valid state, e.g., that a Time data structure won't hold the value of 25:61:61. The conditions checked by asserts are:

• Preconditions, which assure that the caller keeps its contract,

• Postconditions, which assure that the callee keeps its contract, and

• Invariants, which assure that the data structure always holds some property after the function returns. An invariant is a condition that is a precondition and a postcondition.

Unit tests are useful for telling you about the external behavior of the module. Your Stack may have a consistent state after the push() method is called, but if the size of the stack doesn't increase by three after it is called three times, then that is an error. (For example, the trivial case where the incorrect push() implementation only checks the asserts and exits.)

Strictly speaking, the major difference between asserts and unit tests is that unit tests have test data (values to get the program to run), while asserts do not. That is, you can execute your unit tests automatically, while you cannot say the same for assertions. For the sake of this discussion I've assumed that you are talking about executing the program in the context of higher-order function tests (which execute the whole program, and do not drive modules like unit tests). If you are not talking about automated function tests as the means to "see real inputs", then clearly the value lies in automation, and thus the unit tests would win. If you are talking about this in the context of (automated) function tests, then see below.

There can be some overlap in what is being tested. For example, a Stack's postcondition may actually assert that the stack size increases by one. But there are limits to what can be performed in that assert: Should it also check that the top element is what was just added?

For both, the goal is to increase quality. For unit testing, the goal is to find bugs. For assertions, the goal is to make debugging easier by observing invalid program states as soon as they occur.

Note that neither technique verifies correctness. In fact, if you conduct unit testing with the goal to verify the program is correct, you will likely come up with uninteresting test that you know will work. It's a psychological effect: you'll do whatever it is to meet your goal. If your goal is to find bugs, your activities will reflect that.

Both are important, and have their own purposes.

[As a final note about assertions: To get the most value, you need to use them at all critical points in your program, and not a few key functions. Otherwise, the original source of the problem might have been masked and hard to detect without hours of debugging.]

Answer 2

When talking about assertions, keep in mind that they can be turned off at the flick of a switch.

Example of a very bad assertion:

char *c = malloc(1024);
assert(c != NULL);

Why is this bad? Because no error checking is done if that assertion is skipped due to something like NDEBUG being defined.

A unit test would (likely) just segfault on the code above. Sure, it did its job by telling you that something went wrong, or did it? How likely is malloc() to fail under the test?

Assertions are for debugging purposes when a programmer needs to imply that no 'normal' event would cause the assertion to fire. malloc() failing is, indeed a normal event, thus should never be asserted.

There are many other cases where assertions are used instead of adequately handling things that might go wrong. This is why assertions get a bad reputation, and why languages like Go don't include them.

Unit tests are designed to tell you when something you changed broke something else. They are designed to save you (in most cases) from going through every feature of the program (however, testers are important for releases) every time you make a build.

There really isn't any distinct correlation between the two, other than both of them telling you that something went wrong. Think of an assert as a breakpoint in something you are working on, without having to use a debugger. Think of a unit test as something that tells you if you broke something that you aren't working on.

Answer 3

They are both tools used to help improve the overall quality of the system you are building. A lot depends on the language you're using, the type of application you are building, and where your time is best spent. Not to mention you have a few schools of thought about it.

To start, if you are using a language without an assert keyword, you can't use asserts (at least not in the way we are talking here). For a long time, Java did not have an assert keyword, and a number of languages still don't. Unit testing then becomes quite a bit more important. In some languages the assertions are only run when a flag is set (again with Java here). When the protections are not always there, it's not a very useful feature.

There is a school of thought that says if you are "asserting" something, you might as well write an if/throw meaningful exception block. This thought process comes from many asserts placed in the beginning of a method to ensure all values are in bounds. Testing your preconditions is a very important part of having an expected postcondition.

Unit testing is extra code that has to be written and maintained. To many this is a drawback. However, with the current crop of unit test frameworks out there, you can generate a larger number of test conditions with comparatively little code. Parameterized tests and "theories" will perform the same test with a large number of data samples which can uncover some hard to find bugs.

I personally find I get more mileage with unit testing than sprinkling asserts, but that is because of the platforms I develop on most of the time (Java/C#). Other languages have more robust assertion support, and even "Design by Contract" (see below) to provide even more guarantees. If I were working with one of these languages, I might use the DBC more than the unit testing.

http://en.wikipedia.org/wiki/Design_by_contract#Languages_with_native_support