14

I'm reading "Learning Python" and have come across the following:

User-defined exceptions can also signal nonerror conditions. For instance, a search routine can be coded to raise an exception when a match is found instead of returning a status flag for the caller to interpret. In the following, the try/except/else exception handler does the work of an if/else return-value tester:

class Found(Exception): pass

def searcher():
    if ...success...:
        raise Found()            # Raise exceptions instead of returning flags
    else:
        return

Because Python is dynamically typed and polymorphic to the core, exceptions, rather than sentinel return values, are the generally preferred way to signal such conditions.

I've seen this sort of thing discussed multiple times on various forums, and references to Python using StopIteration to end loops, but I can't find much in the official style guides (PEP 8 has one offhand reference to exceptions for flow control) or statements from developers. Is there anything official that states this is best practice for Python?

This (Are exceptions as control flow considered a serious antipattern? If so, Why?) also has several commenters state that this style is Pythonic. What is this based on?

TIA

22

The general consensus “don't use exceptions!” mostly comes from other languages and even there is sometimes outdated.

  • In C++, throwing an exception is very costly due to “stack unwinding”. Every local variable declaration is like a with statement in Python, and the object in that variable may run destructors. These destructors are executed when an exception is thrown, but also when returning from a function. This “RAII idiom” is an integral language feature and is super important to write robust, correct code – so RAII versus cheap exceptions was a tradeoff that C++ decided towards RAII.

  • In early C++, a lot of code was not written in an exception-safe manner: unless you actually use RAII, it is easy to leak memory and other resources. So throwing exceptions would render that code incorrect. This is no longer reasonable since even the C++ standard library uses exceptions: you can't pretend exceptions don't exist. However, exceptions are still an issue when combining C code with C++.

  • In Java, every exception has an associated stack trace. The stack trace is very valuable when debugging errors, but is wasted effort when the exception is never printed, e.g. because it was only used for control flow.

So in those languages exceptions are “too expensive” to be used as control flow. In Python this is less of an issue and exceptions are a lot cheaper. Additionally, the Python language already suffers from some overhead that makes the cost of exceptions unnoticeable compared to other control flow constructs: e.g. checking if a dict entry exists with an explicit membership test if key in the_dict: ... is generally exactly as fast as simply accessing the entry the_dict[key]; ... and checking if you get a KeyError. Some integral language features (e.g. generators) are designed in terms of exceptions.

So while there is no technical reason to specifically avoid exceptions in Python, there is still the question whether you should use them instead of return values. The design-level problems with exceptions are:

  • they are not at all obvious. You can't easily look at a function and see which exceptions it may throw, so you don't always know what to catch. The return value tends to be more well-defined.

  • exceptions are non-local control flow which complicates your code. When you throw an exception, you don't know where the control flow will resume. For errors that can't be immediately handled this is probably a good idea, when notifying your caller of a condition this is entirely unnecessary.

Python culture is generally slanted in favour of exceptions, but it's easy to go overboard. Imagine a list_contains(the_list, item) function that checks whether the list contains an item equal to that item. If the result is communicated via exceptions that is absolutely annoying, because we have to call it like this:

try:
  list_contains(invited_guests, person_at_door)
except Found:
  print("Oh, hello {}!".format(person_at_door))
except NotFound:
  print("Who are you?")

Returning a bool would be much clearer:

if list_contains(invited_guests, person_at_door):
  print("Oh, hello {}!".format(person_at_door))
else:
  print("Who are you?")

If the function is already supposed to return a value, then returning a special value for special conditions is rather error-prone, because people will forget to check this value (that's probably the cause of 1/3 of the problems in C). An exception is usually more correct.

A good example is a pos = find_string(haystack, needle) function that searches for the first occurrence of the needle string in the `haystack string, and returns the start position. But what if they haystack-string does not contain the needle-string?

The solution by C and mimicked by Python is to return a special value. In C this is a null pointer, in Python this is -1. This will lead to surprising results when the position is used as a string index without checking, especially as -1 is a valid index in Python. In C, your NULL pointer will at least give you a segfault.

In PHP, a special value of a different type is returned: the boolean FALSE instead of an integer. As it turns out this isn't actually any better due to the implicit conversion rules of the language (but note that in Python as well booleans can be used as ints!). Functions that do not return a consistent type are generally considered very confusing.

A more robust variant would have been to throw an exception when the string can't be found, which makes sure that during normal control flow it is impossible to accidentally use the special value in place of an ordinary value:

 try:
   pos = find_string(haystack, needle)
   do_something_with(pos)
 except NotFound:
   ...

Alternatively, always returning a type that can't be used directly but must first be unwrapped can be used, e.g. a result-bool tuple where the boolean indicates whether an exception occurred or if the result is usable. Then:

pos, ok = find_string(haystack, needle)
if not ok:
  ...
do_something_with(pos)

This forces you to handle problems immediately, but it gets annoying very quickly. It also prevents you from chaining function easily. Every function call now needs three lines of code. Golang is a language that thinks this nuisance is worth the safety.

So to summarize, exceptions are not entirely without problems and can definitively be overused, especially when they replace a “normal” return value. But when used to signal special conditions (not necessarily just errors), then exceptions can help you to develop APIs that are clean, intuitive, easy to use, and difficult to misuse.

  • Thanks for the in-depth answer. I'm coming from a background of other languages like Java where "exceptions are only for exceptional conditions," so this is new to me. Are there any Python core devs that have ever stated this the way they have with other Python guidelines like EAFP, EIBTI, etc. (on a mailing list, blog post, etc.) I'd like to be able to cite something official if another dev/boss asks. Thanks! – J B Jun 18 '17 at 22:25
  • By overloading the fillInStackTrace method of your custom exception class to just return immediately, you can make it very cheap to raise, as it is stack-walking that is expensive and this is where it is done. – John Cowan Feb 18 '18 at 22:09
  • Your first bullet in your intro was confusing at first. Maybe you could change it to something like "Exception handling code in modern C++ is zero-cost; but throwing an exception is expensive"? (for lurkers: stackoverflow.com/questions/13835817/…) [edit: edit proposed] – phresnel Apr 13 '18 at 7:49
  • Note that for dictionary lookup operations using a collections.defaultdict or my_dict.get(key, default) makes the code much clearer than try: my_dict[key] except: return default – Steve Barnes Jan 26 at 9:03
10

NO! - not in general - exceptions are not considered good flow control practice with the exception of single class of code. The one place where exceptions are considered a reasonable, or even better, way to signal a condition is generator or iterator operations. These operations can return any possible value as a valid result so a mechanism is needed to signal a finish.

Consider reading a binary file of stream one byte at a time - absolutely any value is a potentially valid result but we still need to signal an end of file. So we have a choice, return two values, (the byte value & a valid flag), every time or raise an exception when there is no more to do. In the two cases the consuming code can look like:

# Using validity flag
valid, val = readbyte(source)
while valid:
    processbyte(val)
    valid, val = readbyte(source)
tidy_up()

alternatively:

# With exceptions
try:
   val = readbyte(source)
   processbyte(val) # Note if a problem occurs here it will also raise an exception
except Exception: # Use a specific exception here!
   tidy_up()

But this has, since PEP 343 was implemented & back ported, all been neatly wrapped up in the with statement. The above becomes, the very pythonic:

with open(source) as input:
    for val in input.readbyte(): # This line will raise a StopIteration exception an call input.__exit__()
        processbyte(val) # Not called if there is nothing read

In python3 this became:

for val in open(source, 'rb').read():
     processbyte(val)

I strongly urge you to read PEP 343 which gives the background, rationale, examples, etc.

It is also usual to use an exception to signal the end of processing when using generator functions to signal the finish.

I would like to add that your searcher example is almost certainly backwards, such functions should be generators, returning the first match on the first call, then substituent calls returning the next match, and raising a NotFound exception when there are no more matches.

  • You say: "NO! - not in general - exceptions are not considered good flow control practice with the exception of single class of code". Where is the rationale for this emphatic statement? This answer seems to focus narrowly on the iteration case. There are many other case where people might think to use exceptions. What is the problem with doing so in those other cases? – Daniel Waltrip Jan 25 at 23:20
  • @DanielWaltrip I see far too much code, in various programming languages, where exceptions are used, often far too widely, when a simple if would be better. When it comes to debugging & testing or even reasoning about your codes behavior it is better not to rely on exceptions - they should be the exception. I have seen, in production code, a calculation that returned via a throw/catch rather than a simple return this meant that when calculation hit a divide by zero error it returned a random value rather than crashing where the error happened this caused several hours of debugging. – Steve Barnes Jan 26 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.