Skip to content

Custom Iterators#

An iterator is an object that implements __iter__() (returning self) and __next__() (raising StopIteration when exhausted). Iterators are the protocol behind for loops, comprehensions, and many stdlib tools. Custom iterators let you traverse bespoke data sources lazily — files, graphs, infinite streams — without materializing everything in memory.

How to use this page

Read after Data Structures and Intermediate Syntax and Structures (comprehensions). Contrast with generators (yield-based) in sibling pages under this folder. Pairs with Custom Containers and Map, Filter and Reduce.

At a glance
Track Python Advanced → Iterarators, Generators and Coroutines
Sections 12 major topics
Outline Use the right-hand TOC to jump

Topics: Iterator protocol recap · Iterable vs iterator · Hand-rolled iterator class · Iterator over a custom collection · Iterator over file chunks (lazy I/O) · Infinite iterators · StopIteration and generator migration (Python 3.7+) · __getitem__ legacy protocol · … (+4 more)

  1. Iterator protocol recap
  2. Iterable vs iterator
  3. Hand-rolled iterator class
  4. Iterator over a custom collection
  5. Iterator over file chunks (lazy I/O)
  6. Infinite iterators
  7. StopIteration and generator migration (Python 3.7+)
  8. __getitem__ legacy protocol
  9. Composing with itertools
  10. Custom iterator + dunder methods
  11. Thread safety
  12. Performance notes

Iterator protocol recap#

nums = iter([10, 20, 30])  # list_iterator — built-in iterator
next(nums)  # 10
next(nums)  # 20
next(nums)  # 30
next(nums)  # StopIteration
Method Contract
__iter__() Return an iterator object (usually self)
__next__() Return next value or raise StopIteration

for x in obj: roughly does:

it = iter(obj)       # calls obj.__iter__()
while True:
    try:
        x = next(it)  # calls it.__next__()
    except StopIteration:
        break

Iterable vs iterator#

Iterable Iterator
Defines __iter__ → fresh iterator __iter__ + __next__
Consumable once? No — can iterate repeatedly Yes — one pass
Example list, dict, custom collection list_iterator, file line iterator 
data = [1, 2, 3]
it1 = iter(data)
it2 = iter(data)  # fresh iterator — OK

list(it1)  # [1, 2, 3]
list(it1)  # [] — exhausted
list(it2)  # [1, 2, 3] — independent

Hand-rolled iterator class#

class Countdown:
    def __init__(self, start: int) -> None:
        self.current = start

    def __iter__(self) -> "Countdown":
        return self

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

list(Countdown(3))  # [3, 2, 1]

State lives on self — why iterators are single-pass.

Iterator over a custom collection#

Separate container (iterable) from iterator when you need multiple concurrent traversals:

class BookShelf:
    def __init__(self, titles: list[str]) -> None:
        self._titles = titles

    def __iter__(self):
        return BookIterator(self._titles)


class BookIterator:
    def __init__(self, titles: list[str]) -> None:
        self._titles = titles
        self._index = 0

    def __iter__(self):
        return self

    def __next__(self) -> str:
        if self._index >= len(self._titles):
            raise StopIteration
        title = self._titles[self._index]
        self._index += 1
        return title

for book in BookShelf(["1984", "Dune"]):
    print(book)

Generators simplify this

A __iter__ that yields is usually shorter — but explicit iterator classes teach the protocol and allow complex state (backtracking, peek).

Iterator over file chunks (lazy I/O)#

from pathlib import Path

class ChunkReader:
    def __init__(self, path: Path, chunk_size: int = 1024) -> None:
        self.path = path
        self.chunk_size = chunk_size

    def __iter__(self):
        with self.path.open("rb") as f:
            while True:
                chunk = f.read(self.chunk_size)
                if not chunk:
                    break
                yield chunk

Technically a generator inside __iter__, which returns a generator iterator — idiomatic and memory-efficient. File patterns: File Handling.

Infinite iterators#

class Fibonacci:
    def __init__(self) -> None:
        self.a, self.b = 0, 1

    def __iter__(self):
        return self

    def __next__(self) -> int:
        value = self.a
        self.a, self.b = self.b, self.a + self.b
        return value

from itertools import islice
list(islice(Fibonacci(), 8))
# [0, 1, 1, 2, 3, 5, 8, 13]

Never list(infinite_iterator) without bounds — memory blowup or hang.

StopIteration and generator migration (Python 3.7+)#

Inside generator functions, return value sets StopIteration.value (rarely used). Do not raise StopIteration manually inside generators — use return instead (PEP 479).

# class-based — raise StopIteration in __next__
# generator-based — fall off end or bare return
def naturals():
    n = 1
    while True:
        yield n
        n += 1

__getitem__ legacy protocol#

Before the iterator protocol was universal, classes could implement sequential __getitem__ with IndexError:

class OldStyle:
    def __init__(self, data):
        self.data = data

    def __getitem__(self, index):
        if index >= len(self.data):
            raise IndexError
        return self.data[index]

list(OldStyle([1, 2, 3]))  # still works

Prefer __iter__ / __next__ for new code — clearer and composable with iter().

Composing with itertools#

import itertools

class Range:
    def __init__(self, start: int, stop: int, step: int = 1) -> None:
        self.start, self.stop, self.step = start, stop, step

    def __iter__(self):
        return iter(range(self.start, self.stop, self.step))

# pipeline
data = Range(0, 100, 2)
evens_squared = (x * x for x in data if x % 4 == 0)
sum(evens_squared)

Functional patterns: Map, Filter and Reduce.

Custom iterator + dunder methods#

Containers often implement both iteration and sizing:

class Stack[T]:
    def __init__(self) -> None:
        self._items: list[T] = []

    def push(self, item: T) -> None:
        self._items.append(item)

    def __len__(self) -> int:
        return len(self._items)

    def __iter__(self):
        # top-to-bottom iteration without mutating
        return reversed(self._items)

s = Stack[int]()
s.push(1)
s.push(2)
list(s)  # [2, 1]

See Dunder or Magic Methods and Custom Containers.

Thread safety#

Standard iterators are not thread-safe. Concurrent __next__ on the same iterator races on internal index state. Options:

  • One iterator per thread
  • Lock around __next__
  • Immutable snapshots (tuple(iterable))

Performance notes#

Approach Memory When
list materialization O(n) Need reuse, indexing
Custom iterator O(1) extra Streaming, large data
Generator (yield) O(1) extra Same, less boilerplate

Interview traps (quick reference)#

Trap What goes wrong Safe approach
Reusing exhausted iterator Empty second loop Call iter() again on iterable
__iter__ returns new iterator? Infinite recursion if returns wrong object Return self only for iterator class
Infinite iterator in list() Hang / OOM itertools.islice
Iterator vs iterable in isinstance Wrong mental model collections.abc.Iterable / Iterator
Mutating collection during iteration Skipped elements / RuntimeError Iterate copy or mark safe

Mental model checklist#

  1. What two methods define the iterator protocol?
  2. Why can you iterate a list twice but not a list_iterator?
  3. When should the iterable return a new iterator object from __iter__?
  4. What does StopIteration signal to a for loop?
  5. When is a class-based iterator preferable to a generator?

What's next#

Topic Page
Custom containers Custom Containers
Dunder methods Dunder or Magic Methods
map / filter / reduce Map, Filter and Reduce
Higher-order functions Higher Order Functions
File streaming File Handling