9

PHP, C#, Python and likely a few other languages have a yield keyword that is used to create generator functions.

In PHP: http://php.net/manual/en/language.generators.syntax.php

In Python: https://www.pythoncentral.io/python-generators-and-yield-keyword/

In C#: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/yield

I am concerned that as a language feature/facility, yield breaks some conventions. One of them is what I'd refer to is "certainty". It is a method that returns a different result every time you call it. With a regular non-generator function you can call it and if it is given the same input, it will return the same output. With yield, it returns different output, based on its internal state. Thus if you randomly call the generating function, not knowing its previous state, you cannot expect it to return a certain result.

How does a function like this fit into the language paradigm? Does it actually break any conventions? Is it a good idea to have and use this feature? (to give an example of what's good and what's bad, goto was once a feature of many languages and still is, but it is considered harmful and as such was eradicated from some languages, such as Java). Do programming language compilers/interpreters have to break out of any conventions to implement such a feature, for example, does a language have to implement multi-threading for this feature to work, or can it be done without threading technology?

  • 4
    yield is essentially a state engine. It's not meant to return the same result every time. What it will do with absolute certainty is return the next item in an enumerable each time it is invoked. Threads are not required; you need a closure (more or less), in order to maintain the current state. – Robert Harvey Dec 19 '17 at 23:34
  • 1
    As to the quality of "certainty," consider that, given the same input sequence, a series of calls to the iterator will yield exactly the same items in exactly the same order. – Robert Harvey Dec 19 '17 at 23:40
  • 4
    I'm not sure where most of your questions are coming from since C++ does not have a yield keyword like Python does. It has a static method std::this_thread::yield(), but that's not a keyword. So the this_thread would prepend almost any call to it, making it fairly obvious it's a library feature just for yielding threads, not a language feature about yielding control flow in general. – Ixrec Dec 20 '17 at 1:17
  • link updated to C#, one for C++ removed – Dennis Dec 20 '17 at 15:31
16

Caveats first - C# is the language I know best, and while it has a yield that seems to be very similar to other languages' yield, there may be subtle differences I am unaware of.

I am concerned that as a language feature/facility, yield breaks some conventions. One of them is what I'd refer to is "certainty". It is a method that returns a different result every time you call it.

Poppycock. Do you really expect Random.Next or Console.ReadLine to return the same result every time you call them? How about Rest calls? Authentication? Get Item off a collection? There are all sorts of (good, useful) functions that are impure.

How does a function like this fit into the language paradigm? Does it actually break any conventions?

Yes, yield plays really badly with try/catch/finally, and is disallowed (https://blogs.msdn.microsoft.com/ericlippert/2009/07/16/iterator-blocks-part-three-why-no-yield-in-finally/ for more info).

Is it a good idea to have and use this feature?

It's certainly a good idea to have this feature. Things like C#'s LINQ is really nice - lazily evaluating collections provides a big performance benefit, and yield allows that sort of thing to be done in a fraction of the code with a fraction of the bugs that a hand-rolled iterator would.

That said, there's not a ton of uses for yield outside of LINQ style collection processing. I've used it for validation processing, schedule generation, randomization, and a few other things, but I expect that most developers have never used it (or misused it).

Do programming language compilers/interpreters have to break out of any conventions to implement such a feature, for example, does a language have to implement multi-threading for this feature to work, or can it be done without threading technology?

Not exactly. The compiler generates a state machine iterator that keeps track of where it stopped so that it can start there again the next time it is called. The process for code generation does something akin to Continuation Passing Style, where the code after the yield is pulled into its own block (and if it has any yields, another sub-block, and so on). That's a well known approach used more often off in Functional Programming and also shows up in C#'s async/await compilation.

No threading is needed, but it does require a different approach to code generation in most compilers, and does have some conflict with other language features.

All in all though, yield is a relatively low impact feature that really helps with a specific subset of problems.

  • I've never used C# seriously but this yield keyword is similar to coroutines, yes, or something different? If so I wish I had one in C! I can think of at least some decent sections of code that would have been so much easier to write with such a language feature. – user204677 Dec 20 '17 at 1:52
  • 2
    @DrunkCoder - similar, but with some limitations, as I understand it. – Telastyn Dec 20 '17 at 1:58
  • 1
    You also wouldn't want to see yield misused. The more features a language has, the more likely you will find a program written badly in that language. I'm not sure if the right approach to writing an approachable language is to throw it all at you and see what sticks. – Neil Dec 20 '17 at 13:06
  • 1
    @DrunkCoder: it is a limited version of semi-coroutines. Actually, it is treated as a syntactic pattern by the compiler which gets expanded into a series of method calls, classes, and objects. (Basically, the compiler generates a continuation object that captures the current context in fields.) The default implementation for collections is a semi-coroutine, but by overloading the "magic" methods the compiler uses, you can actually customize the behavior. For example, before async/await was added to the language, someone implemented it using yield. – Jörg W Mittag Dec 20 '17 at 15:36
  • 1
    @Neil It is generally possible to misuse virtually any programming language feature. If what you say were true, then it would be much harder to program badly using C than Python or C#, but this isn't the case as those languages have a lot of tools which protect programmers from many of the mistakes which are very easy to make with C. In reality, the cause of bad programs is bad programmers - it's quite a language-agnostic problem. – Ben Cottrell Dec 22 '17 at 16:40
12

Is having a generator language facility such as yield a good idea?

I'd like to answer this from a Python perspective with an emphatic yes, it's a great idea.

I'll start by addressing some questions and assumptions in your question first, then demonstrate the pervasiveness of generators and their unreasonably usefulness in Python later.

With a regular non-generator function you can call it and if it is given the same input, it will return the same output. With yield, it returns different output, based on its internal state.

This is false. Methods on objects can be thought of as functions themselves, with their own internal state. In Python, since everything is an object, you can actually get a method from an object, and pass around that method (which is bound to the object it came from, so it remembers its state).

Other examples include deliberately random functions as well as input methods like the network, file system, and terminal.

How does a function like this fit into the language paradigm?

If the language paradigm supports things like first class functions, and the generators support other language features like the Iterable protocol, then they fit in seamlessly.

Does it actually break any conventions?

No. Since it's baked into the language, the conventions are built around and include (or require!) the use of generators.

Do programming language compilers/interpreters have to break out of any conventions to implement such a feature

As with any other feature, the compiler simply needs to be designed to support the feature. In the case of Python, functions are already objects with state (such as the default arguments and function annotations).

does a language have to implement multi-threading for this feature to work, or can it be done without threading technology?

Fun fact: The default Python implementation doesn't support threading at all. It features a Global Interpreter Lock (GIL), so nothing is actually running concurrently unless you've spun up a second process to run a different instance of Python.


note: examples are in Python 3

Beyond Yield

While the yield keyword can be used in any function to turn it into a generator, it's not the only way to make one. Python features Generator Expressions, a powerful way to clearly express a generator in terms of another iterable (including other generators)

>>> pairs = ((x,y) for x in range(10) for y in range(10) if y >= x)
>>> pairs
<generator object <genexpr> at 0x0311DC90>
>>> sum(x*y for x,y in pairs)
1155

As you can see, not only is the syntax clean and readable, but built-in functions like sum accept generators.

With

Check out the Python Enhancement Proposal for the With statement. It's very different than you might expect from a With statement in other languages. With a little help from the standard library, Python's generators work beautifully as context managers for them.

>>> from contextlib import contextmanager
>>> @contextmanager
def debugWith(arg):
        print("preprocessing", arg)
        yield arg
        print("postprocessing", arg)


>>> with debugWith("foobar") as s:
        print(s[::-1])


preprocessing foobar
raboof
postprocessing foobar

Of course, printing things is about the most boring thing you can do here, but it does show visible results. More interesting options include auto-management of resources (opening and closing files/streams/network connections), locking for concurrency, temporarily wrapping or replacing a function, and decompressing then recompressing data. If calling functions is like injecting code into your code, then with statements is like wrapping parts of your code in other code. However you use it, it's a solid example of an easy hook into a language structure. Yield-based generators aren't the only way to create context managers, but they certainly are a convenient one.

For and Partial Exhaustion

For loops in Python work in an interesting way. They have the following format:

for <name> in <iterable>:
    ...

First, the expression I called <iterable> is evaluated to get an iterable object. Second, the iterable has __iter__ called on it, and the resulting iterator is stored behind the scenes. Subsequenty, __next__ is called on the iterator to get a value to bind to the name you put in <name>. This step repeats until the call to __next__ throws a StopIteration. The exception is swallowed by the for loop, and execution continues from there.

Coming back to generators: when you call __iter__ on a generator, it just returns itself.

>>> x = (a for a in "boring generator")
>>> id(x)
51502272
>>> id(x.__iter__())
51502272

What this means is you can separate iterating over something from the thing you want to do with it, and change that behavior mid way through. Below, note how the same generator is used in two loops, and in the second it starts executing from where it left off from the first.

>>> generator = (x for x in 'more boring stuff')
>>> for letter in generator:
        print(ord(letter))
        if letter > 'p':
                break


109
111
114
>>> for letter in generator:
        print(letter)


e

b
o
r
i
n
g

s
t
u
f
f

Lazy Evaluation

One of the down sides to generators as compared with lists is the only thing you can access in a generator is the next thing that comes out of it. You can't go back and as for a previous result, or jump ahead to a later one without going through the intermediate results. The up side of this is a generator can take up almost no memory compared to its equivalent list.

>>> import sys
>>> sys.getsizeof([x for x in range(10000)])
43816
>>> sys.getsizeof(range(10000000000))
24
>>> sys.getsizeof([x for x in range(10000000000)])
Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    sys.getsizeof([x for x in range(10000000000)])
  File "<pyshell#10>", line 1, in <listcomp>
    sys.getsizeof([x for x in range(10000000000)])
MemoryError

Generators can also be lazily chained.

logfile = open("logs.txt")
lastcolumn = (line.split()[-1] for line in logfile)
numericcolumn = (float(x) for x in lastcolumn)
print(sum(numericcolumn))

The first, second, and third lines just define a generator each, but don't do any real work. When the last line is called, sum asks numericcolumn for a value, numericcolumn needs a value from lastcolumn, lastcolumn asks for a value from logfile, which then actually reads a line from the file. This stack unwinds until sum gets its first integer. Then, the process happens again for the second line. At this point, sum has two integers, and it adds them together. Note that the third line hasn't been read from the file yet. Sum then goes on requesting values from numericcolumn (totally oblivious to the rest of the chain) and adding them, until numericcolumn is exhausted.

The really interesting part here is that the lines are read, consumed, and discarded individually. At no point is the whole file in memory all at once. What happens if this log file is, say, a terabyte? It just works, because it only reads one line at a time.

Conclusion

This is not a complete review of all uses of generators in Python. Notably, I skipped infinite generators, state machines, passing values back in, and their relationship to coroutines.

I do believe it's sufficient to demonstrate that you can have generators as a cleanly integrated, useful language feature.

6

If you are used to classic OOP languages, generators and yield may seem jarring because mutable state is captured at the function level rather than the object level.

The question of "certainty" is a red herring though. It is usually called referential transparency, and basically means the function always return the same result for the same arguments. As soon as you have mutable state, you lose referential transparency. In OOP, objects often have mutable state, which means the result of method call does not just depend on the arguments, but also the internal state of the object.

The question is where to capture the mutable state. In a classic OOP, mutable state exist at the object level. But if a language support closures, you may have mutable state at the function level. For example in JavaScript:

function getCounter() {
   var cnt = 1;
   return function(){ return cnt++; }
}
var counter = getCounter();
counter() --> 1
counter() --> 2

In short, yield is natural in a language which support closures, but would be out of place in a language like older version of Java where mutable state only exist at the object level.

  • I suppose if language features had a spectrum, yield would be as far away from functional as could be. That's not necessarily a bad thing. OOP was once very fashionable, and again later functional programming. I suppose the danger of it really amounts to mixing and matching features like yield with a functional design that makes your program behave in unexpected ways. – Neil Dec 20 '17 at 13:10
0

In my opinion, it is not a good feature. It is a bad feature, primarily because it needs to be taught very carefully, and everyone teaches it wrong. People use the word "generator," equivocating between the generator function and the generator object. The question is: just who or what is doing the actual yielding?

This is not merely my opinion. Even Guido, in the PEP bulletin in which he rules on this, admits that the generator function is not a generator but a "generator factory."

That's kind of important, don't you think? But reading 99% of the documentation out there, you'd get the impression that the generator function is the actual generator, and they tend to ignore the fact you also need a generator object.

Guido considered replacing "def" for "gen" for these functions and said No. But I'd argue that wouldn't have been enough anyway. It should really be:

def make_gen(args)
    def_gen foo
        # Put in "yield" and other beahvior
    return_gen foo

protected by gnat Nov 23 '18 at 22:00

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.