3

I read Bertrand Meyer's paper on design by contract yesterday and it is not very clear for me what is the relationship between DbC and testing, since it appears that without testing I cannot be certain all the assertions were hit at some point.

Consider a square root function that has preconditions defined that verify the function is never called with a negative number.

Somewhere in the application code the function is used, consider something like

if(n >= 0) {
   // a bunch of code
}
else {
   // some code
   square_root(n)
}

The programmer that added the call to square_root(n) did not consider that it would be called with a negative number (evidently a bug was just introduced).

Obviously, if I run the program in a way that it reaches this branch, with assertions on, we would get an exception reported and we could identify the problem immediately. If we make sure to hit every single call of square_root with any possible category of value it could take, and we don't get any assertions broken, then we know our program is correct and, in production code, we could safely run it without assertions and assume it keeps its reliability properties.

But this has the implication that the code in question needs to be tested. And so, in my example above of the square root function being called with a negative number, if we don't test it extensively in a way that every assertion is hit, and if we switch off assertions in production, then I may end up with a horrible production bug.

And so, it appears that DbC without testing is somewhat useless to ensure my program is correct and reliable. On other hand, if I write unit tests to make sure the square root function is called only with the right kind of arguments, then what's the point of the assertions if my tests are doing all the work of making sure my code is right?

So, what I'm trying to understand here is what is the relationship between DbC and unit testing, and how unit testing does not make DbC meaningless since my tests are a substitute for the assertions in the code.

What am I missing?

13
  • 5
    That paper is 30 years old. Perhaps some more updated information might be useful? For example, Wikipedia states: "Design by contract does not replace regular testing strategies, such as unit testing, integration testing and system testing. Rather, it complements external testing with internal self-tests that can be activated both for isolated tests and in production code during a test-phase." ... Sep 30, 2021 at 15:57
  • ... "The advantage of internal self-tests is that they can detect errors before they manifest themselves as invalid results observed by the client. This leads to earlier and more specific error detection." Sep 30, 2021 at 15:57
  • If your question is "should I test every contract assertion I write," I'd say the answer is "no". In general, don't write trivial unit tests. Forgetting to put in an assertion is morally equivalent to forgetting to put in a field, and you don't write tests for that. Sep 30, 2021 at 16:09
  • But, @RobertHarvey, if you write an assertion, but you don't have a test for it, what is the purpose of the assertion other than perhaps documentation. I'm failing to see its value. How do you justify that?
    – edalorzo
    Sep 30, 2021 at 16:13
  • 1
    Because if your assertion fails, your development environment stops running. Ergo, you've trapped the problem before it gets to production. Assertions are a form of Code Contract; see here. Sep 30, 2021 at 16:16

4 Answers 4

6

DbC isn't necessarily realized with "assertions which are deactivated in production". Assumed there are no insane performance requirements, contract checks should stay in the code when it is operated in production. That way, they will help to detect bugs which have slipped through the net of unit tests, and may become apparent first time with production data. A violated contract will make the program "crash early", and not sweep the issue under the rug, which could otherwise lead to nasty, subsequent faults with a much harder-to-find root cause.

But even if one uses only debug level assertions for DbC, they can be helpful to find bugs not covered by unit tests - specificially during integration tests and end-to-end tests. The latter kind of tests will usually not point you to the specific unit which has a defect. When an end-to-end test fails, in a suffciently complex system, you may know that something went wrong, but have a really hard time to find the root cause for it.

The situation changes a lot when your code contains assertions or contract checks: those will make it more likely that a problem will show up a lot earlier, ideally nearby the code section which is responsible for it (it might not be directly the buggy code itself, but I know from first-hand experience that such assertions can make it a lot easier to spot the heart of such an issue).

3
  • In those cases where assertions are enforced at all times, it appears DbC is not very different from offensive programming, right?
    – edalorzo
    Oct 1, 2021 at 16:51
  • @edalorzo: I had to look up offensive programming, but no: DbC with runtime checks is always form of offensive programming, regardless if the constraints are "enforced at all times", or just in "debug mode".
    – Doc Brown
    Oct 1, 2021 at 18:21
  • 1
    @edalorzo: after reading the question again, I think there was a potential misunderstanding. I guess you conflated "DbC" with "(debug level) assertions". But DbC isn't necessarily "debug only", and not all assertions are DbC. What you correctly observed is that any kind of such assertions are only helpful during testing and debugging, not in production, so one needs to make test to gain some benefit from them. But those don't have to be unit tests - they con be manual tests, integration tests or other kind of tests, and the resulting benefit should be clear from my answer.
    – Doc Brown
    Oct 2, 2021 at 7:15
3

If we are writing a contract for the square root function we would have to define the pre and post conditions.

An obvious precondition would be that the input is positive. As a precondition its the responsibility of the calling code to check it, assuming the bug is that its not checked, then the function is free, not to check and to throw an error, get into an infinite loop, or anything else.

If a pre-condition was not specified then we have to think about the post-conditions. Again with a square root obvious post-conditions would be around how to return a complex number, perhaps we return it as two floats. in which case we are fine and can compute the sqrt of negative numbers. Perhaps we must return a real number and our post-condition is to return a real number which when multiplied by itself will be within some error of the input.

Well then we put this check into the function and throw an error when it is not satisfied.

In either situation we are doing something different from writing tests, we are asserting conditions of the contract.

If we have both the outlined pre and post conditions, we have still never tested what happens when -1, or any other number is passed into the function, but we do know that errors will be thrown if

  • The function attempts to return a value that when squared doesn't equal the input
  • The calling code attempts to pass a negative number to the function.

In this fashion the code is reliable in that it fails reliably when the pre or post conditions are not met

1
  • It is curious that the answers describe something that sounds more like defensive/offensive programming with guards enforced in code at all times more than assertions that can be switched on/off. I wonder in what ways this forms of DbC differ from defensive programming than.
    – edalorzo
    Oct 1, 2021 at 16:57
3

You are focusing on dynamic testing, which means executing the code to see if it works according to the specification in the form of contracts. Partial testing deals with most common use cases. It would tell that in the expected situations, the code works without problems. Exhaustive testing covers both all branches of execution and all feasible values of program variables. That would guarantee that the program always works as expected. This approach, however, is hardly applicable in practice due to the state space explosion.

Static testing provides some guarantees without running a program, i.e. at compile time. Like with dynamic testing, it can be partial, usually implemented by static analyzers that are often incomplete and unsound, but can catch some issues, including the one you suggest as an example - this is a matter of having appropriate analysis rules. The rules can involve contracts in addition to regular code. The static counterpart for exhaustive dynamic testing - verification - guarantees that the program never violates any contract. It does not rely on any test code, just on what is in the program. In fact, additional tests would not help here: they demonstrate that the program works as expected in certain cases, but verification goes far beyond it - it demonstrates that the program always does that.

Verification tools utilizing contracts to show program correctness at compile time is not science fiction. AutoProof does it for Eiffel. Another example is Dafny that integrates verification deeply into the development process. Both rely on Design by contract.

1
  • Oh, I can see, a static analyzer takes advantage of the contract formal definitions to detect where in the code the contract is broken, without actually needing any tests. That’s really good, provided that there’s such tool for my language.
    – edalorzo
    Oct 1, 2021 at 17:04
2

Your question is well-substantiated, but in my opinion your conclusion is... less accurate than it could be, because of some minor implicit assumptions in your underlying reasoning.

First, let's tackle the main question at face value:

Is design by contract useful without unit testing?

This question is like asking "is having a car useful if you don't have a driver's license?" You might think it is clearly not (how would you use it???), but when you start considering that (a) you don't need a license to drive on private property and (b) you might be able to afford to hire a chauffeur or even (c) you just like collecting cars, makes you realize that your initial response is not accounting for all possible use cases.

The real answer to both your question and my analogy is: "Yes there is a benefit. Although in a lot of cases there might not be, there are some cases in which this still makes sense."

Other non-testing benefits of design by contract is the ability to substitute implementations while minimizing the impact on the surrounding codebase of having to do so. If I have an IPersonRepository, I can make any number of concrete data storage implementations, and I can easily swap out one implementation for another. Without a contract, I would not have been able to do this unless I (a) overwrote a new implementation on top of an old one or (b) rewired the references to the old implementation all across the codebase.


I also want to address the smaller assumptions and points you make in your well-bodied question, because I think it's valuable to re-evaluate your assumptions and decisions, and it might help you in understanding why the world isn't already the way you're proposing it should be.


Multiple points of failure are hard to deal with.

Consider a square root function that has preconditions defined that verify the function is never called with a negative number.

The example you provide relies on square_root not validating its input correctly. That's just a bad implementation then. This issue is created by two sources: someone offended (they passed a negative number when they shouldn't have), and someone failed to defend (no validation on positive numbers).

In general, and I'm aware that this is an over-generalization for the sake of argument, you will find that most error-detection systems allow for a single point of failure, but often not two or more.
The point I'm trying to make here is that a lot of "safety" procedures, both error-detection algorithms and good practice guidelines alike, cannot handle when multiple actors act out of line at the same time, as it becomes impossible to know who is a reliable source of what should be the case, and who is not.


Release. Troubleshoot. Expand test suite. Repeat.

The continuation of that realization is that testing is not about preventing issues, it's about making sure to write a detection method for any issue you've encountered so it doesn't happen again.

I cannot stress this enough: It is unrealistic to expect to never let any mistake slip through the cracks, and it is unproductive to hold yourself or anyone else to that standard.

If you write tests for anything that could ever possibly go wrong, you'd hardly ever manage to finish a project and release it. Instead, a much more pragmatic approach to take is:

  • Write tests for obvious failures
  • Accept that you probably forgot some fringe cases that were not clearly in need of testing
  • For each bug that occurs, work out what kind of test would've detected this issue
  • Add that test to the suite to ensure that you don't have to deal with the same bug again in the future.

Over time, your codebase's test suite will become stronger and stronger, holding back regressions as much as possible.

You'll also find that over time, your ability to preemptively spot potential bugs (and thus what tests to write from the start, see the first bullet point) gets better and better, thus reducing the frequency of needing to troubleshoot (third bullet point)

The example you used is a square root calculation. I would've assumed that testing for negative numbers is an obvious test case. But clearly, in your example scenario you failed to test for this during the first pass, and now you're dealing with a bug.

So you find the source, realize that square_root really should have been rejecting negative numbers, and write a test that confirms this.

At the same time, you can also write a test for the consumer of square_root to verify if it ever tries to send a negative number to square_root (assuming that square_root is an external dependency that you can mock for the purpose of your test, and inspect what value your component under test tried to pass into it).
While this is a nice extra test to have, it's less urgent than the square_root test itself. If you only do one of these, make sure it's the square_root test that you do.


Unproductive overkill (subjective)

since it appears that without testing I cannot be certain all the assertions were hit at some point

Frame challenge: is this really necessary, though?

enter image description here

If we make sure to hit every single call of square_root with any possible category of value it could take, and we don't get any assertions broken, then we know our program is correct and, in production code, we could safely run it without assertions and assume it keeps its reliability properties.

Or, to sum it up:

enter image description here

This is, in my opinion, overkill. It's trying to swat a fly with a swatter the size of your house.

Funnily enough, the square_root example works both pro and con here, but mostly con.

Pro: If you assume that n is an integer value, not a decimal one, and because square_root only takes in a single integer value, it's not beyond the realm of possibility to run a test for every possible integer value, since the range of options is finite and computable with modern technology.
The corollary here is that you could do this for any method which takes in a single integer value.

Con You're only really focused on exceptions, but that's not the only thing you should be testing. You should also test if the correct value is being returned, but I doubt you'll want to start listing the all (input, expectedOutput) pairs for every possible integer value you could pass into it.

Con: When dealing with multiple inputs, the processing cost of all possible permutations skyrockets. For example, GetHypotenuse(a,b) (which inherently uses a square root calculation wouldn't have double the amount of tests needed, it has amount of tests needed to run. This is where things start getting very hairy.

Con: When dealing with decimal inputs, it becomes significantly harder to iterate over all possible values and test with them. Not impossible, per se, but significantly more cumbersome.

Con: For this particular example, the developer of square_root could have avoided the entire debacle by setting n to be an unsigned value type.

Con: For square root calculations, the boundaries of what is valid/invalid input is very clear: no negative numbers. End of sentence.

That last con is the most important. Any developer with a lick of contextual awareness is able to deduce that you don't need to test every possible input value, you just have to test with at least one negative value.

If you've already tested with input -1 and the test passes, is there really a purpose to testing input -2? Are you reasonably expecting the square_root method to somehow distinguish between different negative values and selectively properly handle one and not the other? I very much doubt it.

Especially when we get into much more complex input types such as strings, complex objects, datetimes, ... your "test all the values" approach falls apart and becomes both unfeasible and unhelpful. The intention is good, but you really need to take the size and complexity constraints into more consideration than your currently have.

1
  • You mentioned hypotenuse… There will obviously be rounding errors. But you want hyp(x,y) = hyp(y,x) always, and you want that a' >= a, b' >= b to imply hyp(a', b') >= hyp(a, b) for all positive , b.
    – gnasher729
    Oct 1, 2021 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.