8

Python iterators can be very memory efficient. Should I always prefer to use generators instead of just lists? In what situations should I prefer a plain array?

For example instead of this:

emails = [user.email for user in users]

should I prefer this?:

emails = (user.email for user in users)

Note: I do mean 'generators' not 'iterators'.

  • Unclear what help you need. Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it’s hard to tell what problem you are trying to solve or what aspect of your approach needs to be corrected or explained. See the How to Ask page for help clarifying this question. – gnat Feb 17 '15 at 21:19
  • @gnat would the question rephrased as "when should I use iterators or lists in python?" be fine? – Florian Margaine Feb 17 '15 at 21:25
  • @FlorianMargaine that would put it at risk of being closed as too broad I'm afraid – gnat Feb 17 '15 at 21:27
  • 2
    @FlorianMargaine That would be a good start, but it's still kind of a weak question. Where's the author's research? Surely he can think of a few differences. Asking to enumerate all the situations where you might want to use one or the other isn't that great. – Doval Feb 17 '15 at 21:28
  • 2
    +1 It may be a slightly unclear question, but when you're just learning to use these structures, your questions are going to vaguely formed and worded. I learned a lot from the answers. – Racheet Feb 18 '15 at 12:59
7

The main drawback of generators is they can only be traversed in one direction. There's no going back to a previous value. You also can't share them. There are many cases where that can easily be accounted for, or even where it is preferable, but there are also many cases where it isn't. Sorting, for example.

That's why a lot of times you'll see generators used for the initial processing stages of a large amount of data, then once it's filtered into a subset and mapped into a nice format, it gets put into a long-lasting concrete data structure for further use. That way you don't spend the memory allocation, cache misses, and garbage collection costs for large intermediate arrays you'll immediately throw away.

3

You've got iterators confused with generators.

Your first example is a list iterator expression while the second is a generator expression. The key difference is that the generator creates each member of the given collection lazily (as needed) rather than eagerly (at once, whether needed or not). You can define your own generators by using yield rather than return.

As far as usage goes, you want an iterator when you need the list (or dictionary or whatever) for something in and of itself. You'd use a generator when the collection is incidental to the final product. For example, you might use a generator to get a list of numbers that satisfy some criteria from some other criteria. You don't care about the original list, only the members which fit the criteria. Thus, you use a generator to only get those numbers.

A more concrete example would be finding the angles of triangles whose sides are a given length. We don't want all the poor, deformed failed triangles, only the strong, healthy successful triangles. Thus we'd use a generator.

This Stack Overflow post goes into more detail.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.