In a spare-time project of mine, I implemented RSA public-key cryptosystem.

Because the official PKCS#1 standard specify key formats in terms of ASN.1 syntax and DER/BER coding, which is a coding with variable-length encoding of lengths, I had to design a 2-pass interface to accommodate its encoding, where

  • in the 1st pass, coders estimates the necessary buffer length to hold the encoded keys, and
  • perform the encoding in the 2nd pass.

That's a while ago, and since then I've implemented elliptic-curve cryptosystems, which are much easier to implement (except the ECDSA signature, which is an ASN.1 SEQUENCE of 2 INTEGERs). Now, when I review my code for the ASN.1 coders, I realized the 2nd pass of my RSA key coders aren't robust enough - the 2nd pass just assumed buffer is big enough to hold the key, and turn into undefined behavior when the assumption doesn't hold.

The question that I want to ask is: How robust should an (implementation of an) interface be? How to balance the responsibility of the caller and of the callee, and the efficiency of the code?

  • It depends what you're doing and how critical failure is. Commented Oct 1, 2023 at 14:56
  • @PhilipKendall Hi there, hi-rep fellow. I'm thinking from the perspective of a programmer that's going to use my API, and I want to make it hard to mis-use my API. What's your perspective when you mention criticality?
    – DannyNiu
    Commented Oct 1, 2023 at 15:07
  • 1
    @DannyNiu If your code failing means that the nuclear reactor blows up, you should be orders of magnitude more careful than if your code failing means that a button is blue instead of green. Commented Oct 1, 2023 at 15:33
  • 2
    @DaveG or even better, write safety critical code in a language where buffer overflows are (close to) impossible. 99+% of code doesn't need to be in C/C++ Commented Oct 1, 2023 at 15:34
  • 2
    Robustness is a nonfunctional requirement. Your requirements depend solely on the audience for which you write your software.
    – Doc Brown
    Commented Oct 2, 2023 at 6:29

3 Answers 3


Correct use of your API must work.

That is so fundamental for building on it, that debating it doesn't even make sense.
Yes, there are systems where a reasonably small chance of blowing up in the users face is tolerated for cost-related reasons, but that doesn't apply here.

It implies that it must be possible to use it correct, and it should be documented (how else to know what is correct).
As an example of a well-known API failing the first test, look at gets(). No need to search for the second.

Additionally, correct use should be as straight-forward and efficient as possible, and incorrect use look that bad.

Admittedly, sometimes ease-of-use and efficiency/adaptability are at odds, so there could be a low-level and a high-level API, but both must still work.

Applying that, estimating a buffer-length for the user to provide only works if it is reasonably tight but never too small, and efficient.

Otherwise, consider letting the callee allocate it, maybe using a callback provided by the caller, maybe by also providing an API for the proper deallocator.


efficiency of the code?

If efficiency issues are unavoidably leaking into your interface consider offering more than one interface.

Efficiency is one of those things whose importance is hard to predict. Users have different needs.

It’s not uncommon to have one interface with training wheels that makes it easy to use and debug and another that is meant for those with a critical need for speed who are willing to do the work to get it.


Examples of undefined behaviour exist in programming and have a purpose, but it should be avoided if possible.

I think it would be acceptable for your library to have documentation saying for example that the behaviour of calling the second pass without enough buffer space is undefined. You then put the burden on the caller to check or deal with the situation. Maybe there is a performance gain to be had and you want to let the caller decide on their risk levels.

However, this would be unusual. If you can simply check whether the buffer space is sufficient, or detect when its has been exceeded, you could throw an error, stop, or return some partial result.

You can then defined the behaviour of the function with no downsides and you code is 'better'

If the result is undefined AND the undefinedness is undocumented, I think that is just a bug.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.