Iterators & the Iterator Protocol
15 questions found
What is the difference between an iterable and an iterator in Python?
Beginner
An iterable is any object implementing __iter__() that returns an iterator, such as lists, tuples, and dicts -- you can loop over it repeatedly. An iterator is an object implementing both __iter__() (returning itself) and __next__() (returning the next value or raising StopIteration) -- it's stateful and exhausted after one full pass.
nums = [1, 2, 3] # iterable
it = iter(nums) # iterator
print(next(it)) # 1
print(next(it)) # 2
Real-world example
A file object is an iterator over its lines -- you can only read through it once before needing to reopen or seek back to the start.
Common follow-ups: Can an object be both an iterable and an iterator?;What happens if you call next() on an exhausted iterator?
Comprehensions & Generators;Magic Methods & Operator Overloading
How do you implement a custom iterator by defining __iter__ and __next__?
Intermediate
A custom iterator class defines __next__() to return the next value or raise StopIteration when exhausted, and __iter__() to return self so the object works directly in for-loops and with the built-in iter() function.
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
self.current -= 1
return self.current + 1
for n in Countdown(3):
print(n) # 3 2 1
Real-world example
A custom paginated API client implements __next__ to fetch and yield the next page of results lazily as the caller iterates.
Common follow-ups: Why must __iter__ return self on an iterator?;How does this differ from a generator-based approach?
Iterators & the Iterator Protocol;Comprehensions & Generators
What exception signals the end of iteration, and how does a for-loop use it internally?
Beginner
StopIteration signals that an iterator has no more values. A for-loop repeatedly calls next() on the iterator behind the scenes and automatically catches StopIteration to end the loop cleanly, without the exception propagating visibly to your code.
it = iter([1, 2])
while True:
try:
print(next(it))
except StopIteration:
break
Real-world example
Understanding this lets you manually drive an iterator with next() and a try/except when you need finer control than a for-loop provides, such as peeking ahead.
Common follow-ups: What happens if StopIteration is raised inside a generator function in Python 3.7+?;How is StopAsyncIteration different?
Async Generators & Async Context Managers;Exception Handling
How does Python's iter() function work with a sentinel value as a second argument?
Intermediate
iter(callable, sentinel) repeatedly calls callable with no arguments and yields each result until the result equals sentinel, at which point iteration stops. This is useful for turning polling-style APIs (like reading fixed-size chunks or reading until a marker) into a clean iterator.
with open('data.bin', 'rb') as f:
for chunk in iter(lambda: f.read(1024), b''):
process(chunk)
Real-world example
A file-reading loop uses iter(f.read_chunk, '') to process a stream in fixed-size blocks until an empty read signals end-of-file.
Common follow-ups: What other APIs commonly use the sentinel pattern?;How does this compare to a while-loop with a manual read?
File I/O & Context Managers;Iterators & the Iterator Protocol
What is the difference between __iter__ returning self versus returning a separate iterator object?
Advanced
If a class's __iter__ returns self, the object is both iterable and iterator, meaning it can only be iterated once -- a second for-loop over the same instance starts from wherever the first left off (likely exhausted). If __iter__ returns a fresh iterator object each time, the class is a reusable iterable that supports multiple independent iteration passes, which is the more robust and expected design for container-like classes.
class ReusableRange:
def __init__(self, n):
self.n = n
def __iter__(self):
return iter(range(self.n)) # fresh iterator each call
r = ReusableRange(3)
print(list(r), list(r)) # both give [0, 1, 2]
Real-world example
A custom Dataset class in a data pipeline returns a new iterator from __iter__ each epoch so training can loop over the same dataset multiple times.
Common follow-ups: Why do generator objects only support single-pass iteration?;How would you make a generator-based class reusable?
Comprehensions & Generators;Data Types & Structures
How does the itertools module complement the iterator protocol?
Intermediate
itertools provides fast, memory-efficient building blocks that operate on the iterator protocol: infinite iterators (count, cycle, repeat), combinatoric generators (product, permutations, combinations), and terminating iterators (chain, islice, groupby, takewhile). They let you compose complex lazy pipelines without materializing intermediate lists.
import itertools
for combo in itertools.combinations([1,2,3], 2):
print(combo) # (1,2) (1,3) (2,3)
for x in itertools.islice(itertools.count(10), 3):
print(x) # 10 11 12
Real-world example
A test suite uses itertools.product to generate every combination of browser, OS, and screen resolution for cross-browser test cases without nested loops.
Common follow-ups: What's the memory advantage of itertools over building lists?;How does groupby require sorted input?
functools & Functional Programming Tools;Comprehensions & Generators
How would you implement a lazy, chainable iterator pipeline (like filter-then-map) from scratch?
Advanced
You can chain iterators by wrapping one iterator's __next__ around another's, pulling values on demand and applying transformations lazily rather than eagerly. Each stage only computes a value when the next stage requests it, keeping memory usage constant regardless of input size.
class MapIter:
def __init__(self, it, fn):
self.it, self.fn = it, fn
def __iter__(self):
return self
def __next__(self):
return self.fn(next(self.it))
pipeline = MapIter(iter([1,2,3]), lambda x: x*10)
print(list(pipeline)) # [10, 20, 30]
Real-world example
A log-processing tool chains custom filter and transform iterators over a multi-gigabyte log file so only one line is ever held in memory at a time.
Common follow-ups: How do generator expressions achieve the same laziness more concisely?;What's the performance trade-off of custom iterator classes vs generators?
Comprehensions & Generators;Concurrency (asyncio/threading/multiprocessing)
What does the built-in next() function's optional default argument do?
Beginner
next(iterator, default) returns default instead of raising StopIteration when the iterator is exhausted. This avoids needing a try/except block when you just want a fallback value for an empty or finished iterator.
empty = iter([])
print(next(empty, 'no items')) # 'no items' instead of StopIteration
Real-world example
A search function uses next((x for x in items if x.match), None) to get the first matching item or None without writing an explicit loop.
Common follow-ups: How does this pattern replace a manual for-loop with early return?;Is there a performance cost to the default check?
Exception Handling;Comprehensions & Generators
How does Python know whether to treat a for-in target as iterable, and what error occurs if it isn't?
Intermediate
The for-loop calls iter() on the target, which invokes __iter__(). If the object has no __iter__ (and, as a fallback, no __getitem__ supporting integer indexing from 0), Python raises TypeError: 'X' object is not iterable. This is why custom classes need __iter__ (or, historically, __getitem__) to support iteration.
class NoIter:
pass
try:
for x in NoIter():
pass
except TypeError as e:
print(e) # 'NoIter' object is not iterable
Real-world example
A bug report of 'object is not iterable' on a custom API response wrapper leads to adding __iter__ that delegates to an internal list attribute.
Common follow-ups: What is the old-style __getitem__ iteration protocol?;How does isinstance(x, collections.abc.Iterable) check for this?
Magic Methods & Operator Overloading;Data Types & Structures
How does itertools.tee let you iterate over the same iterator multiple times independently?
Advanced
tee(iterable, n) splits a single iterator into n independent iterators that can be advanced separately, internally buffering values that one consumer has read but another hasn't yet. This is essential because a normal iterator is single-pass and consumed once read; tee avoids materializing the whole sequence into a list just to reuse it, though it does use memory proportional to how far apart the consumers get.
import itertools
it = iter([1,2,3,4])
a, b = itertools.tee(it, 2)
print(next(a)) # 1
print(list(b)) # [1,2,3,4]
print(list(a)) # [2,3,4]
Real-world example
A streaming analytics job uses tee to feed the same event stream into both a real-time alerting consumer and a slower batch-aggregation consumer.
Common follow-ups: What happens to memory if one branch lags far behind the other?;When is converting to a list simpler than tee?
functools & Functional Programming Tools;Memory Management & Garbage Collection