When and how should I use exceptions?

Question

The Setting

I often have trouble determining when and how to use exceptions. Let's consider a simple example: suppose I am scraping a webpage, say "http://www.abevigoda.com/", to determine if Abe Vigoda is still alive. To do this, all we need to do is download the page and look for times that the phrase "Abe Vigoda" appears. We return the first appearance, since that includes Abe's status. Conceptually, it will look like this:

def get_abe_status(url):
    # download the page
    page = download_page(url)

    # get all mentions of Abe Vigoda
    hits = page.find_all_mentions("Abe Vigoda")

    # parse the first hit for his status
    status = parse_abe_status(hits[0])

    # he's either alive or dead
    return status == "alive"

Where parse_abe_status(s) takes a string of the form "Abe Vigoda is something" and returns the "something" part.

Before you argue that there are much better and more robust ways of scraping this page for Abe's status, remember that this is just a simple and contrived example used to highlight a common situation I'm in.

Now, where can this code encounter problems? Among other errors, some "expected" ones are:

• download_page might not be able to download the page, and throws an IOError.
• The URL might not point to the right page, or the page is downloaded incorrectly, and so there are no hits. hits is the empty list, then.
• The web page has been altered, possibly making our assumptions about the page wrong. Maybe we expect 4 mentions of Abe Vigoda, but now we find 5.
• For some reasons, hits[0] might not be a string of the form "Abe Vigoda is something", and so it cannot be correctly parsed.

The first case isn't really a problem for me: an IOError is thrown and can be handled by the caller of my function. So let's consider the other cases and how I might handle them. But first, let's assume that we implement parse_abe_status in the stupidest way possible:

def parse_abe_status(s):
    return s[13:]

Namely, it doesn't do any error checking. Now, on to the options:

Option 1: Return None

I can tell the caller that something went wrong by returning None:

def get_abe_status(url):
    # download the page
    page = download_page(url)

    # get all mentions of Abe Vigoda
    hits = page.find_all_mentions("Abe Vigoda")

    if not hits:
        return None

    # parse the first hit for his status
    status = parse_abe_status(hits[0])

    # he's either alive or dead
    return status == "alive"

If the caller receives None from my function, he should assume that there were no mentions of Abe Vigoda, and so something went wrong. But this is pretty vague, right? And it doesn't help the case where hits[0] isn't what we thought it was.

On the other hand, we can put in some exceptions:

Option 2: Using Exceptions

If hits is empty, an IndexError will be thrown when we attempt hits[0]. But the caller shouldn't be expected to handle an IndexError thrown by my function, since he has no idea where that IndexError came from; it could have been thrown by find_all_mentions, for all he knows. So we'll create a custom exception class to handle this:

class NotFoundError(Exception):
    """Throw this when something can't be found on a page."""

def get_abe_status(url):
    # download the page
    page = download_page(url)

    # get all mentions of Abe Vigoda
    hits = page.find_all_mentions("Abe Vigoda")

    try:
        hits[0]
    except IndexError:
        raise NotFoundError("No mentions found.")

    # parse the first hit for his status
    status = parse_abe_status(hits[0])

    # he's either alive or dead
    return status == "alive"

Now what if the page has changed and there are an unexpected number of hits? This isn't catastrophic, as the code may still work, but a caller might want to be extra careful, or he might want to log a warning. So I'll throw a warning:

class NotFoundError(Exception):
    """Throw this when something can't be found on a page."""

def get_abe_status(url):
    # download the page
    page = download_page(url)

    # get all mentions of Abe Vigoda
    hits = page.find_all_mentions("Abe Vigoda")

    try:
        hits[0]
    except IndexError:
        raise NotFoundError("No mentions found.")

    # say we expect four hits...
    if len(hits) != 4:
        raise Warning("An unexpected number of hits.")
        logger.warning("An unexpected number of hits.")

    # parse the first hit for his status
    status = parse_abe_status(hits[0])

    # he's either alive or dead
    return status == "alive"

Lastly, we might find that status isn't either alive or dead. Maybe, for some odd reason, today it turned out to be comatose. Then I don't want to return False, as that implies that Abe is dead. What should I do here? Throw an exception, probably. But what kind? Should I create a custom exception class?

class NotFoundError(Exception):
    """Throw this when something can't be found on a page."""

def get_abe_status(url):
    # download the page
    page = download_page(url)

    # get all mentions of Abe Vigoda
    hits = page.find_all_mentions("Abe Vigoda")

    try:
        hits[0]
    except IndexError:
        raise NotFoundError("No mentions found.")

    # say we expect four hits...
    if len(hits) != 4:
        raise Warning("An unexpected number of hits.")
        logger.warning("An unexpected number of hits.")

    # parse the first hit for his status
    status = parse_abe_status(hits[0])

    if status not in ['alive', 'dead']:
        raise SomeTypeOfError("Status is an unexpected value.")

    # he's either alive or dead
    return status == "alive"

Option 3: Somewhere in between

I think that the second method, with exceptions, is preferable, but I'm not sure if I'm using exceptions correctly within it. I'm curious to see how more experienced programmers would handle this.

Answer 1

The recommendation in Python is to use exceptions to indicate failure. This is true even if you expect failure on a regular basis.

Look at it from the perspective of the caller of your code:

my_status = get_abe_status(my_url)

What if we return None? If the caller doesn't specifically handle the case that get_abe_status failed, it will simply try to continue on with my_stats being None. That may produce a difficult to diagnose bug later on. Even if you do check for None, this code has no clue why get_abe_status() failed.

But what if we raise an exception? If the caller doesn't specifically handle the case, the exception will propagate upward eventually hitting the default exception handler. That may not be the what you want, but its better then introducing a subtle bug elsewhere in the program. Additionally, the exception gives information about what went wrong which is lost in the first version.

From the caller's perspective, its simply more convenient to get an exception than a return value. And that's the python style, to use exceptions to indicate failure conditions not return values.

Some will take a different perspective and argue that you should only use exceptions for cases you never really expect to happen. They argue that normally running running could should not raise any exceptions. One reason that is given for this is that exceptions are grossly inefficient, but that's not actually true for Python.

A couple of points on your code:

try:
    hits[0]
except IndexError:
    raise NotFoundError("No mentions found.")

That's a really confusing way to check for an empty list. Don't induce an exception just to check something. Use an if.

# say we expect four hits...
if len(hits) != 4:
    raise Warning("An unexpected number of hits.")
    logger.warning("An unexpected number of hits.")

You do realize that the logger.warning line will never run right?

Answer 2

The accepted answer deserves to be accepted and answers the question, I write this only to provide a bit of extra background.

One of Python's credos is: it's easier to ask forgiveness than permission. This means that typically you just do things, and if you expect exceptions, you handle them. As opposed to doing if checks before hand to make sure you won't get an exception.

I want to provide an example to show you how dramatic the difference is in mentality from C++/Java. A for loop in C++ typically looks something like:

for(int i = 0; i != myvector.size(); ++i) ...

A way to think about this: accessing myvector[k] where k >= myvector.size() will cause an exception. So you could in principle write this (very awkwardly) as a try-catch.

    for(int i = 0; ; ++i)  {
        try {
           ...
        } catch (& std::out_of_range)
             break

Or something similar. Now, consider what's happening in a python for loop:

for i in range(1):
    ...

How is this working? The for loop takes the result of range(1) and calls iter() on it, grabbing an iterator to it.

b = range(1).__iter__()

Then it calls next on it at each loop iteration, until...:

>>> next(b)
0
>>> next(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

In other words, a for loop in python is actually a try-except in disguise.

As far as the concrete question goes, remember that exceptions stop normal function execution and must be dealt with separately. In Python, you should freely throw them whenever there is no point executing the rest of the code in your function, and/or none of the returns correctly reflect what happened in the function. Note that returning early from a function is different: returning early means you already figured out the answer and don't need the rest of the code to figure out the answer. I am saying that exceptions should be thrown when the answer is not known, and the rest of the code to determine the answer cannot be reasonably run. Now, "correctly reflect" itself, like which exceptions you choose to throw, is all a matter of documentation.

In the case of your particular code, I would say any situation that causes hits to be an empty list should throw. Why? Well, the way your function is setup, there is no way to determine the answer without parsing hits. So if hits is not parseable, either because the URL is bad, or because hits is empty, then the function can't answer the question, and in fact can't even really try to.

In this particular case, I would argue that even if you manage to parse and don't get a reasonable answer (alive or dead), then you should still throw. Why? Because, the function returns a boolean. Returning None is very dangerous to your client. If they do an if check on None, there won't be failure, it will just silently be treated as False. So, your client will basically always have to do an if is None check anyhow if he doesn't want silent failures... so you should probably just throw.

Answer 3

You should use exceptions when something exceptional occurs. That is, something that should not occur given proper use of the application. If it is allowable and expected for the consumer of your method to search for something that will not be found, then "not found" is not an exceptional case. In this case, you should return null or "None" or {}, or something indicating an empty return set.

If, on the other hand, your really do expect the consumers of your method to always (unless they screwed up somehow) find what is being searched for, then not finding it would be an exception and you should go with that.

The key is that exception handling can be expensive--exceptions are supposed to gather information about the state of your application when they occur, such as a stack trace, in order to help folks decipher why they occurred. I don't think that's what you are trying to do.

Answer 4

If I was writing a function

 def abe_is_alive():

I would write it to return True or False in the cases where I am absolutely certain of one or the other, and raise an error in any other case (e.g. raise ValueError("Status neither 'dead' nor 'alive'")). This is because the function calling mine is expecting a boolean, and if I can't provide that with certainty the regular program flow shouldn't continue.

Something like your example of getting a different number of "hits" than expected, I would probably ignore; as long as one of the hits still matches my pattern "Abe Vigoda is {dead|alive}", that's fine. This allows the page to be rearranged but still gets the appropriate information.

Rather than

try:
    hits[0] 
except IndexError:
    raise NotFoundError

I would check explicitly:

if not hits:
    raise NotFoundError

as this tends to be "cheaper" then setting up the try.

I agree with you on IOError; I also wouldn't try to error handle connecting to the website - if we can't, for some reason, this isn't the appropriate place to handle it (as it doesn't help us answer our question) and it should pass out to the calling function.