Python - 'if foo in dict' vs 'try: dict[foo]'

Question

This is less a question about the nature of duck typing and more about staying pythonic, I suppose.

First of all - when dealing with dicts, in particular when the structure of the dict is fairly predictable and a given key is not typically present but sometimes is, I first think of two approaches:

if myKey in dict:
    do_some_work(dict[myKey])
else:
    pass

And of course Ye Olde 'forgiveness vs permission' approach.

try:
    do_some_work(dict[myKey])
except KeyError:
    pass

As a journeyman Python guy, I feel like I see the latter preferred a lot, which only feels odd I guess because in the Python docs try/excepts seem to be preferred when there is an actual mistake, as opposed to an, um… absence of success?

If the occasional dict does not have key in myDict, and it is known that it will not always have that key, is a try/except contextually misleading? This isn't a programming error, it's just a fact of the data - this dict just didn't have that particular key.

This seems particularly important when you look at the try/except/else syntax, which looks to be really useful when it comes to making sure that try isn't catching too many errors. You're able to do something like:

try:
    foo += bar
except TypeError:
    pass
else:
    return some_more_work(foo)

Isn't that going to lead to swallowing all kinds of weird errors that are probably the result of some bad code? The above code might just be preventing you from seeing that you're trying to add 2 + {} and you may never realize that some part of your code has gone horribly wrong. I don't suggest that we should Check All The Types, that's why it's Python and not JavaScript - but again with the context of try/except, it seems like it's supposed to catch the program doing something it shouldn't be doing, instead of enabling it to carry on.

I realize the above example is something of a straw-man argument, and is in fact intentionally bad. But given the pythonic creed of better to ask forgiveness than permission I can't help but feel like it begs the question of where the line in the sand actually is between correct application of if/else vs try/except is, in particular when you know what to expect out of the data you're working with.

I'm not even talking about speed concerns or best practice here, I'm just sort of quietly confused by the perceived Venn diagram of cases where it looks like it could go either way, but people err on the side of a try/except because 'someone somewhere said it was Pythonic'. Have I drawn the wrong conclusions about the application of this syntax?

Answer 1

Use the get method instead:

some_dict.get(the_key, default_value)

...where default_value is the value returned if the_key is not in some_dict. If you omit default_value, None is returned if the key is missing.

In general, in Python, people tend to prefer try/except than checking something first - see the EAFP entry in the glossary. Note that many "test for membership" functions use exceptions behind the scenes.

Answer 2

Is missing key a rule or an exception?

From a pragmatic point of view, throwing/catching is much slower then checking if the key is in table. If missing key is a common occurrence - you should use condition.

Another thing in favor of condition is an else clause - it is much easier to understand when either block if executed, consider that even same exception class can be thrown from several statements in the try block.

Code in except clause should not be a part of the normal flaw (e.g. if key's not there, add it) - it should be handling the error.

Answer 3

In the spirit of being "pythonic" and, specifically, duck typing - the try/except looks almost fragile to me.

All you really want is something with dict-like access, but __getitem__ can be overridden to do something not quite as you expect. For example, using defaultdict from the standard library:

In [1]: from collections import defaultdict

In [2]: foo = defaultdict(int)

In [3]: foo['a'] = 1

In [4]: foo['b']  # Someone did access checking with a try/except exactly as you have in the question
Out[4]: 0

In [5]: 'a' in foo
Out[5]: True

In [6]: 'b' in foo  # We never set it, but now it exists!
Out[6]: True

In [7]: 'c' in foo
Out[7]: False

Because the default return value is Falsy, access checking like that will work most of the time just fine.. It appears to keep the code simpler, too, which was why my co-workers and I had done it for months.

Unfortunately, a few months later, these extra keys actually ended up causing problems and hard-to-track-down bugs, because 0 became a valid value. And sometimes we would use in to check if a key existed, making the code do different things when it should have been identical.

The reason I suggest it looks broken is because in Python's spirit of duck-typing, all you should want is something dict-like, and a defaultdict fits that requirement perfectly, as would any object you might create that inherits from dict. You can't guarantee the caller isn't going to change its implementation later on down the line, so you should do the thing with the least side-effects.

Use the first version, if myKey in mydict.

---

Also, note that this is specifically about the example in the question. The python creed of "Better to ask forgiveness than permission" is largely meant to ensure you actually act correctly when you don't get what you want. For example, checking if a file exists and then trying to read from it - at least 3 things I can think of off the top of my head can go wrong:

• A race condition is that the file might have been deleted/moved/etc
• You don't have read permissions on the file
• The file has been corrupted

The first one you can try to "ask for permission" on, but by nature of a race conditions it wouldn't necessarily always work; the second one is too easy to forget to check; and the third cannot be checked without trying to read the file. In any case, what you do after failing will almost certainly be the same, so in this example, it is better to "ask forgiveness" by using a try/except.