Generators vs Lists in Python: Memory and Lazy Evaluation

Python gives you two fundamentally different approaches to working with sequences of values. A list materialises every element upfront and holds them all in memory simultaneously. A generator produces each value on demand, suspending its own execution between calls and resuming exactly where it left off. The choice between them is not a matter of syntax preference. It determines how much memory a program consumes and whether it can handle inputs of arbitrary size.

What a List Holds in Memory

When Python evaluates a list, it allocates a contiguous block of memory and fills it with references to every object in the collection. A list of one million integers holds one million object references in memory at the same time, regardless of whether the program ever reads them all. The entire sequence exists from the moment the list is created.

numbers = [x * x for x in range(1_000_000)]
# All one million squared values exist in memory right now

This has consequences that are easy to overlook. Filtering a list produces another full list. Transforming a list produces another full list. Chaining three operations across a large dataset creates three full intermediate lists. Each one must be fully allocated before the next step can begin.

What a Generator Does Instead

A generator function is any function that contains a yield expression. Calling it does not execute the function body. It returns a generator iterator, which is an object that implements the iterator protocol. The function body runs only when the next value is requested, advancing until it reaches the next yield, at which point execution is suspended and the yielded value is returned to the caller. Local variables and the entire execution state are preserved between calls.

def squared(n):
    for i in range(n):
        yield i * i

gen = squared(1_000_000)
# Nothing has been computed yet. The function body has not run.

next(gen)  # 0  — executes until the first yield, then suspends
next(gen)  # 1  — resumes from suspension, executes until the next yield
next(gen)  # 4

As the Python documentation states, each yield temporarily suspends processing, remembering the execution state including local variables and pending try-statements. When the generator iterator resumes, it picks up where it left off, in contrast to functions which start fresh on every invocation. Once the function body reaches its end or a return statement, StopIteration is raised and the generator is exhausted.

Generator Expressions and List Comprehensions

The syntactic parallel between generator expressions and list comprehensions makes the distinction concrete. A list comprehension uses square brackets and returns a fully materialised list. A generator expression uses parentheses and returns a lazy iterator.

# List comprehension — all values computed and stored immediately
squares_list = [x * x for x in range(10)]

# Generator expression — no values computed yet
squares_gen = (x * x for x in range(10))

Both support if clauses for filtering. Both support nested for clauses. The only difference is when evaluation happens and where the values live. A generator expression passed directly to a function that consumes an iterable, such as sum or max, does not require the extra parentheses.

total = sum(x * x for x in range(10))  # 285

Measuring the Difference with sys.getsizeof

The memory difference between a list and a generator can be measured directly. sys.getsizeof returns the size of an object in bytes. For a list, this grows with the number of elements. For a generator, it remains fixed because the generator object itself stores only execution state, not the values it will produce.

import sys

n = 1_000_000

list_size = sys.getsizeof([x * x for x in range(n)])
gen_size  = sys.getsizeof(x * x for x in range(n))

print(list_size)  # ~8_448_728 bytes (varies by platform and Python version)
print(gen_size)   # 104 bytes

The generator object is the same size regardless of whether n is ten or ten million. It holds no computed values. The list object grows linearly because it must store a reference to every element it contains.

Note that sys.getsizeof measures the container itself, not the objects it references. The total memory consumed by a list includes the referenced objects as well, making the real difference even larger in practice.

When to Use a Generator

Generators are the right choice when working with large datasets where loading everything into memory at once would be prohibitive or unnecessary. Reading lines from a large file, processing records from a database cursor, or applying a transformation pipeline to streaming data are all cases where each value is consumed once and discarded.

Generators are also the only way to represent infinite sequences. A generator that counts upward without bound, or produces successive terms of a mathematical series, cannot be expressed as a list.

def integers_from(n):
    while True:
        yield n
        n += 1

counter = integers_from(0)
next(counter)  # 0
next(counter)  # 1
next(counter)  # 2
# continues indefinitely

Processing pipelines are another natural fit. Each stage of a pipeline can be a generator expression or generator function that pulls from the previous stage on demand. No intermediate results are fully materialised, and memory usage stays constant regardless of input size.

When to Use a List

A list is the right choice when the program needs to access elements by index, access the same sequence more than once, or when the dataset is small enough that holding it in memory is not a concern. A generator is a one-pass iterator. Once a value has been yielded and the iterator has advanced, that value cannot be retrieved again without recreating the generator from scratch.

gen = (x * x for x in range(5))
list(gen)  # [0, 1, 4, 9, 16]
list(gen)  # []  — exhausted, nothing left to yield

Sorting requires a list. Reversing requires a list. Any operation that needs to know the length upfront, slice a range, or refer back to an earlier element requires that all values exist simultaneously. For these cases, materialising the sequence into a list is not avoidable.

What You Can Do Now

Write a generator function that reads a large text file line by line, strips whitespace, and filters out blank lines, then chain it into a second generator that converts each line to uppercase. Measure the memory each stage consumes using sys.getsizeof and compare it to the equivalent list-based approach.

import sys

def read_lines(filepath):
    with open(filepath) as f:
        for line in f:
            stripped = line.strip()
            if stripped:
                yield stripped

def uppercased(lines):
    for line in lines:
        yield line.upper()

filepath = "your_file.txt"

# Generator pipeline — constant memory regardless of file size
pipeline = uppercased(read_lines(filepath))
print(sys.getsizeof(pipeline))  # fixed, ~104 bytes

# List equivalent — entire file contents held in memory
lines_list = [line.upper() for line in open(filepath) if line.strip()]
print(sys.getsizeof(lines_list))  # grows with file size

# Consume the pipeline
for line in pipeline:
    print(line)

The generator pipeline processes one line at a time: read_lines yields a stripped line, uppercased receives it, transforms it, and yields it forward. At no point does more than one line exist in the pipeline simultaneously. The list comprehension, by contrast, holds every line from the file in memory before the first print runs.