Are exceptions for flow control best practice in Python?

Question

I'm reading "Learning Python" and have come across the following:

User-defined exceptions can also signal nonerror conditions. For instance, a search routine can be coded to raise an exception when a match is found instead of returning a status flag for the caller to interpret. In the following, the try/except/else exception handler does the work of an if/else return-value tester:

class Found(Exception): pass

def searcher():
    if ...success...:
        raise Found()            # Raise exceptions instead of returning flags
    else:
        return

Because Python is dynamically typed and polymorphic to the core, exceptions, rather than sentinel return values, are the generally preferred way to signal such conditions.

I've seen this sort of thing discussed multiple times on various forums, and references to Python using StopIteration to end loops, but I can't find much in the official style guides (PEP 8 has one offhand reference to exceptions for flow control) or statements from developers. Is there anything official that states this is best practice for Python?

This (Are exceptions as control flow considered a serious antipattern? If so, Why?) also has several commenters state that this style is Pythonic. What is this based on?

TIA

Answer 1

The general consensus “don't use exceptions!” mostly comes from other languages and even there is sometimes outdated.

• In C++, throwing an exception is very costly due to “stack unwinding”. Every local variable declaration is like a with statement in Python, and the object in that variable may run destructors. These destructors are executed when an exception is thrown, but also when returning from a function. This “RAII idiom” is an integral language feature and is super important to write robust, correct code – so RAII versus cheap exceptions was a tradeoff that C++ decided towards RAII.

• In early C++, a lot of code was not written in an exception-safe manner: unless you actually use RAII, it is easy to leak memory and other resources. So throwing exceptions would render that code incorrect. This is no longer reasonable since even the C++ standard library uses exceptions: you can't pretend exceptions don't exist. However, exceptions are still an issue when combining C code with C++.

• In Java, every exception has an associated stack trace. The stack trace is very valuable when debugging errors, but is wasted effort when the exception is never printed, e.g. because it was only used for control flow.

So in those languages exceptions are “too expensive” to be used as control flow. In Python this is less of an issue and exceptions are a lot cheaper. Additionally, the Python language already suffers from some overhead that makes the cost of exceptions unnoticeable compared to other control flow constructs: e.g. checking if a dict entry exists with an explicit membership test if key in the_dict: ... is generally exactly as fast as simply accessing the entry the_dict[key]; ... and checking if you get a KeyError. Some integral language features (e.g. generators) are designed in terms of exceptions.

So while there is no technical reason to specifically avoid exceptions in Python, there is still the question whether you should use them instead of return values. The design-level problems with exceptions are:

• they are not at all obvious. You can't easily look at a function and see which exceptions it may throw, so you don't always know what to catch. The return value tends to be more well-defined.

• exceptions are non-local control flow which complicates your code. When you throw an exception, you don't know where the control flow will resume. For errors that can't be immediately handled this is probably a good idea, when notifying your caller of a condition this is entirely unnecessary.

Python culture is generally slanted in favour of exceptions, but it's easy to go overboard. Imagine a list_contains(the_list, item) function that checks whether the list contains an item equal to that item. If the result is communicated via exceptions that is absolutely annoying, because we have to call it like this:

try:
  list_contains(invited_guests, person_at_door)
except Found:
  print("Oh, hello {}!".format(person_at_door))
except NotFound:
  print("Who are you?")

Returning a bool would be much clearer:

if list_contains(invited_guests, person_at_door):
  print("Oh, hello {}!".format(person_at_door))
else:
  print("Who are you?")

If the function is already supposed to return a value, then returning a special value for special conditions is rather error-prone, because people will forget to check this value (that's probably the cause of 1/3 of the problems in C). An exception is usually more correct.

A good example is a pos = find_string(haystack, needle) function that searches for the first occurrence of the needle string in the `haystack string, and returns the start position. But what if they haystack-string does not contain the needle-string?

The solution by C and mimicked by Python is to return a special value. In C this is a null pointer, in Python this is -1. This will lead to surprising results when the position is used as a string index without checking, especially as -1 is a valid index in Python. In C, your NULL pointer will at least give you a segfault.

In PHP, a special value of a different type is returned: the boolean FALSE instead of an integer. As it turns out this isn't actually any better due to the implicit conversion rules of the language (but note that in Python as well booleans can be used as ints!). Functions that do not return a consistent type are generally considered very confusing.

A more robust variant would have been to throw an exception when the string can't be found, which makes sure that during normal control flow it is impossible to accidentally use the special value in place of an ordinary value:

 try:
   pos = find_string(haystack, needle)
   do_something_with(pos)
 except NotFound:
   ...

Alternatively, always returning a type that can't be used directly but must first be unwrapped can be used, e.g. a result-bool tuple where the boolean indicates whether an exception occurred or if the result is usable. Then:

pos, ok = find_string(haystack, needle)
if not ok:
  ...
do_something_with(pos)

This forces you to handle problems immediately, but it gets annoying very quickly. It also prevents you from chaining function easily. Every function call now needs three lines of code. Golang is a language that thinks this nuisance is worth the safety.

So to summarize, exceptions are not entirely without problems and can definitively be overused, especially when they replace a “normal” return value. But when used to signal special conditions (not necessarily just errors), then exceptions can help you to develop APIs that are clean, intuitive, easy to use, and difficult to misuse.

Answer 2

NO! - not in general - exceptions are not considered good flow control practice with the exception of single class of code.

There are a couple of reasons why, in the high reliability & testing worlds at least, exceptions for flow control are not considered a good idea:

1. Exceptions can potentially occur anywhere not just at a return point and may be occurring deep in nested functions.
2. If you are using exceptions to return from function calls then you may end up catching too generally and masking real errors. Consider a maths array function that can raise more than one type of exception to signal that it has finished - if your code uses except Exception: to detect this then divide by zero errors and bounds errors will also be caught as a "Finished" flag.

The one place where exceptions are considered a reasonable, or even better, way to signal a condition is generator or iterator operations. These operations can return any possible value as a valid result so a mechanism is needed to signal a finish.

Consider reading a binary file of stream one byte at a time - absolutely any value is a potentially valid result but we still need to signal an end of file. So we have a choice, return two values, (the byte value & a valid flag), every time or raise an exception when there is no more to do. In the two cases the consuming code can look like:

# Using validity flag
valid, val = readbyte(source)
while valid:
    processbyte(val)
    valid, val = readbyte(source)
tidy_up()

alternatively:

# With exceptions
try:
   val = readbyte(source)
   processbyte(val) # Note if a problem occurs here it will also raise an exception
except Exception: # Use a specific exception here!
   tidy_up()

But this has, since PEP 343 was implemented & back ported, all been neatly wrapped up in the with statement. The above becomes, the very pythonic:

with open(source, 'rb') as input_file:
    for val in input_file.readbyte(): # This line will raise a StopIteration exception an call input.__exit__()
        processbyte(val) # Not called if there is nothing read

In python3 this became:

for val in open(source, 'rb').read():
     processbyte(val)

This hidden use of exceptions is a lot easier to read and comprehend for most people.

I strongly urge you to read PEP 343 which gives the background, rationale, examples, etc.

It is also usual to use an exception to signal the end of processing when using generator functions to signal the finish. Again this is nicely hidden from you when you use:

for value in generator_funcion():
    do_something_with(value)
# Carry on now that you have finished

I would like to add that your searcher example is almost certainly backwards, such functions should be generators, returning the first match on the first call, then substituent calls returning the next match, and raising a NotFound exception when there are no more matches.

Note that these hidden exception uses are using very specific exceptions so that any other exceptions will still be visible. The reason why these hidden exception classes are documented as such is that you may have reasons to override them e.g. if you are reading from a serial port or other device that can sometimes not have a value ready but hasn't actually finished you can deal with this.

Answer 3

I'm attempting to answer this question in a slightly different angle.

• Does Python encourage using exceptions for flow control?
• When to use exceptions in your design, anyway?

Does Python encourage using exceptions for flow control?

I found no official written documentation to support that claim. (To readers who would disagree: please leave comments with links to evidences you found.) The only vaguely-relevant paragraph that I found, is this EAFP term:

EAFP

Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.

Such paragraph merely described that, rather than doing this:

def make_some_noise(speaker):
    if hasattr(speaker, "quack"):
        speaker.quack()

we would prefer this:

def make_some_noise(speaker):
    try:
        speaker.quack()
    except AttributeError:
        logger.warning("This speaker is not a duck")

make_some_noise(DonaldDuck())  # This would work
make_some_noise(DonaldTrump())  # This would trigger exception

or potentially even omitting the try...except:

def make_some_noise(duck):
    duck.quack()

So, the EAFP encourages duck-typing. But it does not encourage using exceptions for flow control.

In what situation you could design your program to emit exceptions?

It is a moot conversation on whether it is a best practice (or an anti-pattern, for that matter) to use exception as control flow. Because, once a design decision is made for a given function, its usage pattern would also be determined, and then the caller would have no choice but to use it that way.

So, let's go back to the fundamentals to see when a function would better produce its outcome via returning a value or via emitting exception(s).

What is the difference between the return value and the exception?

1. Their "blast radius" are different. Return value is only available to the immediate caller; exception can be automatically relayed for unlimited distance until it is caught.

2. Their distribution patterns are different. Return value is by definition one piece of data (even though you could return a compound data type such as a dictionary or a container object, it is still technically one value). The exception mechanism, on the contrary, allows multiple values (one at a time) to be returned via their respective dedicate channel. Here, each except FooError: ... and except BarError: ... block is considered as its own dedicate channel.

Therefore, it is up to each different scenario to use one mechanism that fits well.

• All normal cases should better be returned via return value, because the callers would most likely need to use that return value immediately. The return-value approach also allows nesting layers of callers in a functional programming style. The exception mechanism's long blast radius and multiple channels do not help here. For example, it would be unintuitive if any function named get_something(...) produces its happy path result as an exception. (This is not really a contrived example. There is one practice to implement BinaryTree.Search(value) to use exception to ship the value back in the middle of a deep recursion. The example in this OP's question also contains a similar example.)

• If the caller would likely forget to handle the error sentinel from the return value, it is probably a good idea to use exception's characterist #2 to save caller from its hidden bug. This answer gave a typical non-example of the position = find_string(haystack, needle) whose return value of -1 or null would tend to cause a bug in the caller.

• If the error sentinel would collide with a normal value in the result namespace, it is almost certain to use an exception, because you'd have to use a different channel to convey that error.

• If the normal channel i.e. the return value is already used in the happy-path, AND the happy-path does NOT have sophisicated flow control, you have no choice but to use exception for flow control. People keep talking about how Python uses StopIteration exception for iteration termination, and use it to kind of justify "using exception for flow control". But IMHO this is only a practical choice in a particular situation, it does not generalize and glorify "using exception for flow control".

At this point, if you already make a sound decision on whether your function get_stock_price() would produce only return-value or also raise exceptions, or if that function is provided by an existing library so that its behavior has long be decided, you do not have much choice in writing its caller calculate_market_trend(). Whether to use get_stock_price()'s exception to control the flow in your calculate_market_trend() is merely a matter of whether your business logic requires you to do so. If yes, do it; otherwise, let the exception bubble up to a higher level (this utilizes the characteristic #1 "long blast radius" of exception).

In particular, if you are implementing a middle-layer library Foo and you happen to be making a dependency on lower-level library Bar, you would probably want to hide your implementation detail, by catching all Bar.ThisError, Bar.ThatError, ..., and map them into Foo.GenericError. In this case, the long blast radius is actually working against us, so you might hope "only if library Bar were returning its errors via return values". But then again, that decision has long been made in Bar, so you can just live with it.

All in all, whether to design your software to emit exception is neither a best practice nor an anti-pattern.

Answer 4

Preface

whether to design your software to emit exceptions is neither a best practice nor an anti-pattern. — the immortal wisdom of RayLuo (link)

The above is the best answer I've seen. My answer is intended to supplement RayLuo's with insights not yet covered; none of it is based in things I've seen in the wild - these are just thoughts I've been synthesizing over the years that I want to explore and share. Thanks go to RayLuo for encouraging me to add my thoughts! ♥️

Wishful thinking

The perversity of the Universe tends towards a maximum. — Finagle's law

In programming, we often like to focus on the cases where things go right. When we're designing a new thing, we're asserting something that didn't exist before. It's natural to focus on what that thing is, and not in the uncountably many ways that it can go wrong.

It's also expensive to even think of all those ways that a system can fail, not to mention to encode a description of how to handle those cases when they arise. Our jobs are hard enough as it is, just describing this thing that didn't exist before!

As programmers, we attempt to optimize our code towards a description that matches what we want, and have it take effect by enscribing this description into reality. But reality is a messy place.

The geometry of failure

If you look at most of the access patterns of our programs, they probably look like

>99% success < 1% errors (hopefully?)

but in terms of possible success and failure states, it's probably

>99% errors and < 1% success

through this lens, the success case looks quite exceptional.

Developers don't always handle failure modes...

Let's say you import a module that you didn't write yourself, and your code execution encounters a KeyError from within the foreign module. It doesn't say what it means; it's just a generic KeyError. It could be because of how your code interacted with it, it could be from a misconfigured dependency or expectation of the foreign module, it could be anything.

You write a try/except and start trying to figure things out, but then there's different areas where the KeyError might be coming from, and it could mean something completely different.

...Or success modes...

Now let's look at a different scenario: you import a module that you didn't write yourself, and your code execution encounters a ValueFoundInDatabase exception. Odd. Usually calling a function results in a return value. You need this module though, so you write a try/except for it.

Then you encounter a ValueFoundInCache exception. Huh. You dive deeper and you see that both of these exceptions are sub-classes of SuccessState, which eventually subclasses Exception. You start to trace the trees of exceptions, which is really just a tree of different states the program might terminate in, including a lot of exceptions that derive from FailureState.

You've already written a try/except, and handling the failure states seems easily within reach; it's another except, just like handling the success states, and its implementation merely depends on what precision your application's current form could care to handle: FailureState in general or more specifically: NotFound, TimedOut, InvalidKey, SignatureMismatch...

Suddenly you know you can handle specific edge cases with not much extra work. You don't have to think hard about what they are or how to detect them because the module author already put thought and care into it. Additionally, there's a few extra 'edge' SuccessStates that you can handle with additional flair, because you have easy access to the specific descendants of SuccessState. If you know a successful result came from the cache or the database, you can flow your control accordingly.

The result

Because you're already handling the results with a try/except to handle the SuccessState or its descendants, it's so easy to just add another except for whatever level of other states you prefer to address. That module developer may have wanted to raise your awareness to these other states, and have you handle those edge cases instead of writing a function call and going on your merry way.

The result is also: completely awful for quickly writing functioning code, because you need to try/except anything that has this functionality, and you can't tell which "color" it is in the function coloring sense. In my eyes, it's ugly to look at. But textual code-writing is ugly to look at.

---

Caveat

Another form of the above is to (gasp) return an object that has more information about the state the process terminated in. The object itself could be of different types that represent these states, or more crudely it could have a property called state that taps into something like those exception trees from before (side note: exceptions don't need to be raised, they're just another object).

Long blast radius

There's also the side-benefit/drawback of RayLuo's mention of a "long blast radius". A calling function scope might not be the one to handle the resulting exception; it could skip several layers, which could be a feature in, say, a function in a web server that wants to immediately raise some kind of Http_Response as an exception. This approach is, I suspect, completely terrible. Definitely different than what we're used to. But it might be a feature in the high-flying duck.quack() EAFP example.

Enumeration of states

Sandi Metz has been a proponent of condition-less coding, and I can see why: every if/else is a branching of program state -- and often that results in multiplicative, or even exponential effects. This isn't exactly the same as what I'm showcasing here, but it rhymes.

I'm not saying exceptions are the answer, but I must admit: it's nice to dream of ways to enumerate and integrate all of the relevant states a program is designed to terminate in. This includes:

• variants of success states
• things went great "and by the way" states
• more nuanced "something went wrong, but it's kinda ok?" in-between states
• the huge tree of failure states

Sometimes return value feels a little too simple.