24

I'm reading "Learning Python" and have come across the following:

User-defined exceptions can also signal nonerror conditions. For instance, a search routine can be coded to raise an exception when a match is found instead of returning a status flag for the caller to interpret. In the following, the try/except/else exception handler does the work of an if/else return-value tester:

class Found(Exception): pass

def searcher():
    if ...success...:
        raise Found()            # Raise exceptions instead of returning flags
    else:
        return

Because Python is dynamically typed and polymorphic to the core, exceptions, rather than sentinel return values, are the generally preferred way to signal such conditions.

I've seen this sort of thing discussed multiple times on various forums, and references to Python using StopIteration to end loops, but I can't find much in the official style guides (PEP 8 has one offhand reference to exceptions for flow control) or statements from developers. Is there anything official that states this is best practice for Python?

This (Are exceptions as control flow considered a serious antipattern? If so, Why?) also has several commenters state that this style is Pythonic. What is this based on?

TIA

43

The general consensus “don't use exceptions!” mostly comes from other languages and even there is sometimes outdated.

  • In C++, throwing an exception is very costly due to “stack unwinding”. Every local variable declaration is like a with statement in Python, and the object in that variable may run destructors. These destructors are executed when an exception is thrown, but also when returning from a function. This “RAII idiom” is an integral language feature and is super important to write robust, correct code – so RAII versus cheap exceptions was a tradeoff that C++ decided towards RAII.

  • In early C++, a lot of code was not written in an exception-safe manner: unless you actually use RAII, it is easy to leak memory and other resources. So throwing exceptions would render that code incorrect. This is no longer reasonable since even the C++ standard library uses exceptions: you can't pretend exceptions don't exist. However, exceptions are still an issue when combining C code with C++.

  • In Java, every exception has an associated stack trace. The stack trace is very valuable when debugging errors, but is wasted effort when the exception is never printed, e.g. because it was only used for control flow.

So in those languages exceptions are “too expensive” to be used as control flow. In Python this is less of an issue and exceptions are a lot cheaper. Additionally, the Python language already suffers from some overhead that makes the cost of exceptions unnoticeable compared to other control flow constructs: e.g. checking if a dict entry exists with an explicit membership test if key in the_dict: ... is generally exactly as fast as simply accessing the entry the_dict[key]; ... and checking if you get a KeyError. Some integral language features (e.g. generators) are designed in terms of exceptions.

So while there is no technical reason to specifically avoid exceptions in Python, there is still the question whether you should use them instead of return values. The design-level problems with exceptions are:

  • they are not at all obvious. You can't easily look at a function and see which exceptions it may throw, so you don't always know what to catch. The return value tends to be more well-defined.

  • exceptions are non-local control flow which complicates your code. When you throw an exception, you don't know where the control flow will resume. For errors that can't be immediately handled this is probably a good idea, when notifying your caller of a condition this is entirely unnecessary.

Python culture is generally slanted in favour of exceptions, but it's easy to go overboard. Imagine a list_contains(the_list, item) function that checks whether the list contains an item equal to that item. If the result is communicated via exceptions that is absolutely annoying, because we have to call it like this:

try:
  list_contains(invited_guests, person_at_door)
except Found:
  print("Oh, hello {}!".format(person_at_door))
except NotFound:
  print("Who are you?")

Returning a bool would be much clearer:

if list_contains(invited_guests, person_at_door):
  print("Oh, hello {}!".format(person_at_door))
else:
  print("Who are you?")

If the function is already supposed to return a value, then returning a special value for special conditions is rather error-prone, because people will forget to check this value (that's probably the cause of 1/3 of the problems in C). An exception is usually more correct.

A good example is a pos = find_string(haystack, needle) function that searches for the first occurrence of the needle string in the `haystack string, and returns the start position. But what if they haystack-string does not contain the needle-string?

The solution by C and mimicked by Python is to return a special value. In C this is a null pointer, in Python this is -1. This will lead to surprising results when the position is used as a string index without checking, especially as -1 is a valid index in Python. In C, your NULL pointer will at least give you a segfault.

In PHP, a special value of a different type is returned: the boolean FALSE instead of an integer. As it turns out this isn't actually any better due to the implicit conversion rules of the language (but note that in Python as well booleans can be used as ints!). Functions that do not return a consistent type are generally considered very confusing.

A more robust variant would have been to throw an exception when the string can't be found, which makes sure that during normal control flow it is impossible to accidentally use the special value in place of an ordinary value:

 try:
   pos = find_string(haystack, needle)
   do_something_with(pos)
 except NotFound:
   ...

Alternatively, always returning a type that can't be used directly but must first be unwrapped can be used, e.g. a result-bool tuple where the boolean indicates whether an exception occurred or if the result is usable. Then:

pos, ok = find_string(haystack, needle)
if not ok:
  ...
do_something_with(pos)

This forces you to handle problems immediately, but it gets annoying very quickly. It also prevents you from chaining function easily. Every function call now needs three lines of code. Golang is a language that thinks this nuisance is worth the safety.

So to summarize, exceptions are not entirely without problems and can definitively be overused, especially when they replace a “normal” return value. But when used to signal special conditions (not necessarily just errors), then exceptions can help you to develop APIs that are clean, intuitive, easy to use, and difficult to misuse.

5
  • 1
    Thanks for the in-depth answer. I'm coming from a background of other languages like Java where "exceptions are only for exceptional conditions," so this is new to me. Are there any Python core devs that have ever stated this the way they have with other Python guidelines like EAFP, EIBTI, etc. (on a mailing list, blog post, etc.) I'd like to be able to cite something official if another dev/boss asks. Thanks! – J B Jun 18 '17 at 22:25
  • 1
    By overloading the fillInStackTrace method of your custom exception class to just return immediately, you can make it very cheap to raise, as it is stack-walking that is expensive and this is where it is done. – John Cowan Feb 18 '18 at 22:09
  • Your first bullet in your intro was confusing at first. Maybe you could change it to something like "Exception handling code in modern C++ is zero-cost; but throwing an exception is expensive"? (for lurkers: stackoverflow.com/questions/13835817/…) [edit: edit proposed] – phresnel Apr 13 '18 at 7:49
  • Note that for dictionary lookup operations using a collections.defaultdict or my_dict.get(key, default) makes the code much clearer than try: my_dict[key] except: return default – Steve Barnes Jan 26 '19 at 9:03
  • 1
    Great answer! Minor nitpick: when using C on various embedded platforms, often dereferencing NULL is no problem at all, making it very costly in terms of debugging to find cases where returned pointer values are not checked. And even on x86 dereferencing NULL isn't always problematic (e.g. see lwn.net/Articles/342330). – mvds Apr 26 at 7:38
12

NO! - not in general - exceptions are not considered good flow control practice with the exception of single class of code. The one place where exceptions are considered a reasonable, or even better, way to signal a condition is generator or iterator operations. These operations can return any possible value as a valid result so a mechanism is needed to signal a finish.

Consider reading a binary file of stream one byte at a time - absolutely any value is a potentially valid result but we still need to signal an end of file. So we have a choice, return two values, (the byte value & a valid flag), every time or raise an exception when there is no more to do. In the two cases the consuming code can look like:

# Using validity flag
valid, val = readbyte(source)
while valid:
    processbyte(val)
    valid, val = readbyte(source)
tidy_up()

alternatively:

# With exceptions
try:
   val = readbyte(source)
   processbyte(val) # Note if a problem occurs here it will also raise an exception
except Exception: # Use a specific exception here!
   tidy_up()

But this has, since PEP 343 was implemented & back ported, all been neatly wrapped up in the with statement. The above becomes, the very pythonic:

with open(source) as input:
    for val in input.readbyte(): # This line will raise a StopIteration exception an call input.__exit__()
        processbyte(val) # Not called if there is nothing read

In python3 this became:

for val in open(source, 'rb').read():
     processbyte(val)

I strongly urge you to read PEP 343 which gives the background, rationale, examples, etc.

It is also usual to use an exception to signal the end of processing when using generator functions to signal the finish.

I would like to add that your searcher example is almost certainly backwards, such functions should be generators, returning the first match on the first call, then substituent calls returning the next match, and raising a NotFound exception when there are no more matches.

5
  • 1
    You say: "NO! - not in general - exceptions are not considered good flow control practice with the exception of single class of code". Where is the rationale for this emphatic statement? This answer seems to focus narrowly on the iteration case. There are many other case where people might think to use exceptions. What is the problem with doing so in those other cases? – Daniel Waltrip Jan 25 '19 at 23:20
  • 2
    @DanielWaltrip I see far too much code, in various programming languages, where exceptions are used, often far too widely, when a simple if would be better. When it comes to debugging & testing or even reasoning about your codes behavior it is better not to rely on exceptions - they should be the exception. I have seen, in production code, a calculation that returned via a throw/catch rather than a simple return this meant that when calculation hit a divide by zero error it returned a random value rather than crashing where the error happened this caused several hours of debugging. – Steve Barnes Jan 26 '19 at 8:55
  • Hey @SteveBarnes, the divide by zero error catch example sounds like poor programming (with a too-general catch that doesn't re-raise the exception). – wtr Aug 22 '19 at 16:21
  • Your answer seems to boil down to: A) "NO!" B) there are valid examples of where control flow via exceptions work better than alternatives C) a correction of the question's code to use generators (which use exceptions as control flow, and do so well). While your answer is generally informative, no arguments appear to exist in it to support item A, which, being in bold and seemingly the primary statement, I would expect some support for. If it were supported, I wouldn't downvote your answer. Alas... – wtr Aug 22 '19 at 16:23
  • @wTyeRogers, your comment seems to boil down to: (1) "but I think the answer should be a yes" (2) "Since Steve Barnes did not provide arguments to support his 'No!' statement, I went on to downvote this answer". Hmm, now your comment also became an emphatic statement. wTyeRogers can you provide your answer with arguments to support "using exceptions for flow control is best practice in Python"? – RayLuo Mar 24 at 20:43
2

I'm attempting to answer this question in a slightly different angle.

  • Does Python encourage using exceptions for flow control?
  • When to use exceptions in your design, anyway?

Does Python encourage using exceptions for flow control?

I found no official written documentation to support that claim. (To readers who would disagree: please leave comments with links to evidences you found.) The only vaguely-relevant paragraph that I found, is this EAFP term:

EAFP

Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.

Such paragraph merely described that, rather than doing this:

def make_some_noise(speaker):
    if hasattr(speaker, "quack"):
        speaker.quack()

we would prefer this:

def make_some_noise(speaker):
    try:
        speaker.quack()
    except AttributeError:
        logger.warning("This speaker is not a duck")

make_some_noise(DonaldDuck())  # This would work
make_some_noise(DonaldTrump())  # This would trigger exception

or potentially even omitting the try...except:

def make_some_noise(duck):
    duck.quack()

So, the EAFP encourages duck-typing. But it does not encourage using exceptions for flow control.

In what situation you could design your program to emit exceptions?

It is a moot conversation on whether it is a best practice (or an anti-pattern, for that matter) to use exception as control flow. Because, once a design decision is made for a given function, its usage pattern would also be determined, and then the caller would have no choice but to use it that way.

So, let's go back to the fundamentals to see when a function would better produce its outcome via returning a value or via emitting exception(s).

What is the difference between the return value and the exception?

  1. Their "blast radius" are different. Return value is only available to the immediate caller; exception can be automatically relayed for unlimited distance until it is caught.

  2. Their distribution patterns are different. Return value is by definition one piece of data (even though you could return a compound data type such as a dictionary or a container object, it is still technically one value). The exception mechanism, on the contrary, allows multiple values (one at a time) to be returned via their respective dedicate channel. Here, each except FooError: ... and except BarError: ... block is considered as its own dedicate channel.

Therefore, it is up to each different scenario to use one mechanism that fits well.

  • All normal cases should better be returned via return value, because the callers would most likely need to use that return value immediately. The return-value approach also allows nesting layers of callers in a functional programming style. The exception mechanism's long blast radius and multiple channels do not help here. For example, it would be unintuitive if any function named get_something(...) produces its happy path result as an exception. (This is not really a contrived example. There is one practice to implement BinaryTree.Search(value) to use exception to ship the value back in the middle of a deep recursion. The example in this OP's question also contains a similar example.)

  • If the caller would likely forget to handle the error sentinel from the return value, it is probably a good idea to use exception's characterist #2 to save caller from its hidden bug. This answer gave a typical non-example of the position = find_string(haystack, needle) whose return value of -1 or null would tend to cause a bug in the caller.

  • If the error sentinel would collide with a normal value in the result namespace, it is almost certain to use an exception, because you'd have to use a different channel to convey that error.

  • If the normal channel i.e. the return value is already used in the happy-path, AND the happy-path does NOT have sophisicated flow control, you have no choice but to use exception for flow control. People keep talking about how Python uses StopIteration exception for iteration termination, and use it to kind of justify "using exception for flow control". But IMHO this is only a practical choice in a particular situation, it does not generalize and glorify "using exception for flow control".

At this point, if you already make a sound decision on whether your function get_stock_price() would produce only return-value or also raise exceptions, or if that function is provided by an existing library so that its behavior has long be decided, you do not have much choice in writing its caller calculate_market_trend(). Whether to use get_stock_price()'s exception to control the flow in your calculate_market_trend() is merely a matter of whether your business logic requires you to do so. If yes, do it; otherwise, let the exception bubble up to a higher level (this utilizes the characteristic #1 "long blast radius" of exception).

In particular, if you are implementing a middle-layer library Foo and you happen to be making a dependency on lower-level library Bar, you would probably want to hide your implementation detail, by catching all Bar.ThisError, Bar.ThatError, ..., and map them into Foo.GenericError. In this case, the long blast radius is actually working against us, so you might hope "only if library Bar were returning its errors via return values". But then again, that decision has long been made in Bar, so you can just live with it.

All in all, whether to design your software to emit exception is neither a best practice nor an anti-pattern.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.