Iterators, generators and itertools

The iterator protocol, yield, lazy pipelines, generator expressions, itertools recipes and yield from.

The iterator protocol

An iterable can produce an iterator; an iterator returns the next item from __next__ and raises StopIteration when it is exhausted. A for loop is just that protocol with the exception handled for you.

nums = [1, 2, 3]

it = iter(nums)          # calls nums.__iter__()
next(it)                 # 1
next(it)                 # 2

# what for actually does
while True:
    try:
        value = next(it)
    except StopIteration:
        break

# an iterator is exhausted, and it is also its own iterable
list(it)                 # [3] - continues where we left off
list(it)                 # []  - nothing left

# any object can join the protocol
class Countdown:
    def __init__(self, start):
        self.n = start

    def __iter__(self):
        while self.n > 0:
            yield self.n
            self.n -= 1

print(list(Countdown(3)))   # [3, 2, 1]
print(sum(Countdown(4)))    # 10

• A list holds every value at once; an iterator holds only the position and computes values on demand.
• Once consumed, an iterator cannot be rewound — rebuild it or materialise it into a list when you need two passes.
• Built-ins that take any iterable: sum, max, min, any, all, sorted, zip, list, set, dict.

Generators and lazy pipelines

A function containing yield is a generator function: calling it returns a generator without running any of the body. Execution advances one yield at a time, keeping local state between resumptions.

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:                     # file objects stream already
            yield line.rstrip("\n")

def non_empty(lines):
    for line in lines:
        if line.strip():
            yield line

def field(lines, index, sep=","):
    for line in lines:
        yield line.split(sep)[index]

# the whole pipeline is lazy: no file is opened until the loop runs
rows = field(non_empty(read_lines("data.csv")), 2)
total = 0
for value in rows:                        # one line in memory at a time
    total += len(value)

# generator expression: the same idea in one line, memory-flat
total = sum(len(v) for v in field(non_empty(read_lines("data.csv")), 2))

# a generator can receive values and return a final result
def running_average():
    total, count, avg = 0.0, 0, None
    while True:
        value = yield avg                 # send() resumes here
        total += value
        count += 1
        avg = total / count

gen = running_average()
next(gen)          # prime it: advance to the first yield
gen.send(10)       # 10.0
gen.send(20)       # 15.0

Function pipelines without itertools

You can compose generators with plain functions, but itertools covers the patterns that come up constantly and is implemented in C, so it is faster than hand-rolled equivalents.

from itertools import chain, islice, groupby, accumulate, count, product, combinations, pairwise

calls = chain(day_a, day_b)                  # concatenate iterables lazily
first_10 = list(islice(count(100, 10), 10))  # 100, 110, ... 190
running = list(accumulate([1, 2, 3, 4]))     # [1, 3, 6, 10]
pairs = list(pairwise([1, 2, 3]))            # [(1, 2), (2, 3)]
grid = list(product("AB", [1, 2]))           # [('A',1), ('A',2), ('B',1), ('B',2)]
picks = list(combinations("ABC", 2))         # [('A','B'), ('A','C'), ('B','C')]

records = [
    ("uk", "ada"), ("uk", "alan"), ("us", "grace"),
]
records.sort(key=lambda r: r[0])             # groupby only groups adjacent keys
for region, group in groupby(records, key=lambda r: r[0]):
    print(region, [name for _, name in group])
# uk ['ada', 'alan']
# us ['grace']

# flatten nested iterables and delegate to a sub-iterator
def all_lines(paths):
    for path in paths:
        yield from read_lines(path)          # same as: for line in ...: yield line

• groupby and unique style helpers only merge consecutive equal keys — sort first.
• chain.from_iterable(list_of_lists) flattens one level without building an intermediate list.
• islice, takewhile and dropwhile let you cut a stream without loading it.
• tee clones a stream, but it buffers everything the two copies have not both consumed — measure before using it on large data.
• yield from forwards values, send values and the final return value from the inner generator.

FAQ

Related

Decorators, closures and context managers Object-oriented Python: classes and dunder methods

Last refreshed 2026-09-18.