What is a generator function?

What is a generator function in Python?

A regular function in Python

Here's a regular function:

def negate_all(iterable):
    print("Start")
    negatives = []
    for n in iterable:
        print("Negating", n)
        negatives.append(-n)
    print("End")
    return negatives

What do you think we'll see when we call this function? What do you think we'll get back?

>>> negatives = negate_all([2, 1, 3])

When we call this function, we see a number of things printed out:

>>> negatives = negate_all([2, 1, 3])
Start
Negating 2
Negating 1
Negating 3
End

And we get back the return value of this function:

>>> negatives
[-2, -1, -3]

A generator function in Python

This function is very similar to a regular function:

def negate_all(iterable):
    print("Start")
    for n in iterable:
        print("Negating", n)
        yield -n
    print("End")

But it's not a normal Python function: it's a generator function.

We can only tell it's a generator function by the presence of a yield statement. A yield statement turns a regular function into a generator function.

What do you think we'll see when we call this generator function? What do you think it'll give us back? What will it return to us?

>>> negatives = negate_all([2, 1, 3])

When we call this generator function, we don't see anything printed out. And the thing we get back is a generator object:

>>> negatives = negate_all([2, 1, 3])
>>> negatives
<generator object negate_all at 0x7f5d85fd49e0>

Generator objects are lazy iterables.

Generator objects put themselves on pause

One thing we can do with a generator object is pass it to the built-in next function. Passing a generator object to next will start running the generator function that created it:

>>> negatives = negate_all([2, 1, 3])
>>> x = next(negatives)
Start
Negating 2

The generator function printed out Start and then Negating 2, and then it stopped. It stopped because it hit a yield statement.

It yielded -2 to us:

>>> x
-2

If we call next again (passing in the same generator object) we'll see Negating 1 printed out:

>>> x = next(negatives)
Negating 1

And the generator yielded -1:

>>> x
-1

Each time we pass this generator object to next, the generator function that created it will start up again from where it left off.

Generator objects put themselves on pause to yield an item. Then when you ask them for another item, they'll start running their generator function again until they hit another yield statement.

The next item we get is -3:

>>> x = next(negatives)
Negating 3
>>> x
-3

But then if we pass this generator object to next again, it will hit the end of the function, which will raise a StopIteration exception:

>>> x = next(negatives)
End
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Generator objects are lazy iterables

It's a little bit unusual to see a generator object passed to the built-in next function.

Normally, we loop over generator objects the same way we loop over any other iterable, with a for loop:

>>> negatives = negate_all([2, 1, 3])
>>> for n in negatives:
...     print("Got", n)
...

When we loop over this generator object, we'll see that our generator function is being called in-between our for loop being run:

>>> negatives = negate_all([2, 1, 3])
>>> for n in negatives:
...     print("Got", n)
...
Start
Negating 2
Got -2
Negating 1
Got -1
Negating 3
Got -3
End

So when we ask for the first item, it starts running our generator function.

Start
Negating 2

Then our generator function puts itself on pause to yield that first item to our for loop, which then prints out Got -2 (the first item is -2):

Got -2

Then our for loop asks the generator for another item, which causes the generator function to start running again (unpausing itself). The generator function then prints out Negating 1, and yields -1 to our for loop, which prints out Got -1:

Negating 1
Got -1

And the same thing happens with 3. Our generator function prints out Negating 3, and then we move back to our for loop body, which prints out Got -3:

Negating 3
Got -3

Then finally, when our generator function returns, our for loop ends as well:

End

Generator objects can only be looped over once

Once we've consumed all the items in a generator object, we say that it's been exhausted (meaning it doesn't have anything left in it).

So if we loop over our generator object a second time, we'll see that it's empty now:

>>> for n in negatives:
...     print("Got", n)
...
>>>

A function with yield statements is a generator function

A generator function is a function with one or more yield statements in it.

Unlike regular functions, generator functions return generator objects. Meaning, when you call a generator function, it doesn't run the function. Instead, it gives you back a generator object.

If you loop over that generator object, it will run the function until a yield statement is reached. At that point the generator object will put itself on pause, and yield the next item. This process will continue over and over until you've consumed all the items within the generator.