Generators, Iterators, and Memory Efficiency

📚 Lesson 18 of 20 ⏱️ 55 min

Generators, Iterators, and Memory Efficiency

55 min

Generators are special functions in Python that use the `yield` keyword to produce values one at a time, rather than returning all values at once. This lazy evaluation makes generators extremely memory-efficient for processing large datasets, infinite sequences, or data streams. When a generator function is called, it returns a generator object (an iterator) that can be iterated over. Values are computed on-demand as you iterate, not all at once, which can save significant memory for large datasets.

The iterator protocol consists of two methods: `__iter__()` (returns the iterator itself) and `__next__()` (returns the next value or raises StopIteration). Any object implementing these methods is an iterator. Python's built-in types (lists, tuples, strings, dictionaries) are iterable (they have `__iter__()`), and iterating over them creates iterators. Generators automatically implement the iterator protocol, making them easy to use with for loops, comprehensions, and functions like `sum()`, `max()`, and `list()`.

Generator expressions are like list comprehensions but create generators instead of lists. They use parentheses instead of square brackets: `(x**2 for x in range(10))` creates a generator, while `[x**2 for x in range(10)]` creates a list. Generator expressions are more memory-efficient for large sequences and are perfect for one-time iterations. The `itertools` module provides powerful iterator tools like `chain()` (combine iterators), `islice()` (slice iterators), `cycle()` (infinite repetition), `combinations()`/`permutations()` (combinatorics), and many more.

Memory efficiency is a key benefit of generators. A generator that produces a million values uses constant memory (just the generator object), while a list of a million values uses memory proportional to its size. This makes generators ideal for processing large files, streaming data, infinite sequences, and data pipelines. However, generators can only be iterated once—if you need to iterate multiple times, you must recreate the generator or convert it to a list (defeating the memory benefit).

Common generator patterns include data pipelines (chaining generators together), lazy evaluation (computing values only when needed), infinite sequences (generating values indefinitely), and coroutines (using generators for cooperative multitasking, though `async`/`await` is now preferred). Generators work seamlessly with Python's iteration tools: for loops, comprehensions, `map()`, `filter()`, `zip()`, and more. Understanding when to use generators vs lists is crucial for writing efficient Python code.

Best practices include using generators for large datasets or one-time iterations, using generator expressions when you don't need the full list, being aware that generators are single-use (can't iterate twice), using `itertools` for common iterator patterns, and converting to lists only when necessary. Generators are essential for building scalable Python applications that can handle large amounts of data without running out of memory.

Key Concepts

Generators use yield to produce values one at a time (lazy evaluation).
Iterators implement __iter__() and __next__() methods.
Generator expressions create generators (more memory-efficient than lists).
Generators are single-use—can only be iterated once.
itertools module provides powerful iterator tools.

Learning Objectives

Master

Creating generators with yield for memory-efficient data processing
Understanding the iterator protocol and how it works
Using generator expressions and itertools for efficient pipelines
Building scalable applications that handle large datasets

Develop

Memory-efficient programming thinking
Understanding lazy evaluation and on-demand computation
Designing efficient data processing pipelines

Tips

Use generators for large datasets or one-time iterations.
Use generator expressions when you don't need the full list.
Remember generators are single-use—can't iterate twice.
Leverage itertools for common iterator patterns.

Common Pitfalls

Trying to iterate a generator multiple times, getting empty results.
Converting generators to lists unnecessarily, losing memory benefits.
Not understanding lazy evaluation, causing unexpected behavior.
Using generators when you need random access (generators are sequential).

Summary

Generators enable memory-efficient processing of large datasets.
Generators use lazy evaluation—values computed on-demand.
Generator expressions are more memory-efficient than list comprehensions.
Generators are single-use—iterate once or recreate.
Understanding generators is crucial for scalable Python applications.

Exercise

Create a comprehensive data processing system using generators, iterators, and the itertools module to efficiently process large datasets.

from itertools import chain, islice, tee, groupby, combinations, permutations\nfrom typing import Iterator, List, Dict, Any, Tuple\nimport time\nimport random\n\nclass DataProcessor:\n    """Advanced data processing system using generators and iterators."""\n    \n    def __init__(self):\n        self.processed_count = 0\n        self.memory_usage = 0\n    \n    def generate_large_dataset(self, size: int = 1000000) -> Iterator[int]:\n        """Generate a large dataset without storing it all in memory."""\n        for i in range(size):\n            yield random.randint(1, 1000)\n    \n    def filter_even_numbers(self, data: Iterator[int]) -> Iterator[int]:\n        """Filter even numbers from the dataset."""\n        for item in data:\n            if item % 2 == 0:\n                yield item\n    \n    def transform_data(self, data: Iterator[int]) -> Iterator[Dict[str, Any]]:\n        """Transform numbers into structured data."""\n        for item in data:\n            yield {\n                'value': item,\n                'squared': item ** 2,\n                'is_prime': self._is_prime(item),\n                'factors': list(self._get_factors(item))\n            }\n    \n    def batch_process(self, data: Iterator[Any], batch_size: int = 1000) -> Iterator[List[Any]]:\n        """Process data in batches for memory efficiency."""\n        batch = []\n        for item in data:\n            batch.append(item)\n            if len(batch) >= batch_size:\n                yield batch\n                batch = []\n        if batch:\n            yield batch\n    \n    def parallel_processing(self, data: Iterator[Any], num_streams: int = 4) -> Iterator[Any]:\n        """Split data into multiple streams for parallel processing."""\n        # Create multiple iterators from the same data source\n        iterators = tee(data, num_streams)\n        \n        # Process each stream independently\n        for i, iterator in enumerate(iterators):\n            for item in iterator:\n                yield {\n                    'stream_id': i,\n                    'data': item,\n                    'processed_at': time.time()\n                }\n    \n    def aggregate_results(self, data: Iterator[Dict[str, Any]]) -> Dict[str, Any]:\n        """Aggregate results from processed data."""\n        total_count = 0\n        sum_values = 0\n        prime_count = 0\n        max_value = float('-inf')\n        min_value = float('inf')\n        \n        for item in data:\n            total_count += 1\n            sum_values += item['value']\n            if item['is_prime']:\n                prime_count += 1\n            max_value = max(max_value, item['value'])\n            min_value = min(min_value, item['value'])\n        \n        return {\n            'total_count': total_count,\n            'average_value': sum_values / total_count if total_count > 0 else 0,\n            'prime_count': prime_count,\n            'prime_percentage': (prime_count / total_count * 100) if total_count > 0 else 0,\n            'max_value': max_value,\n            'min_value': min_value\n        }\n    \n    def _is_prime(self, n: int) -> bool:\n        """Check if a number is prime."""\n        if n < 2:\n            return False\n        if n == 2:\n            return True\n        if n % 2 == 0:\n            return False\n        \n        for i in range(3, int(n ** 0.5) + 1, 2):\n            if n % i == 0:\n                return False\n        return True\n    \n    def _get_factors(self, n: int) -> Iterator[int]:\n        """Get all factors of a number."""\n        for i in range(1, int(n ** 0.5) + 1):\n            if n % i == 0:\n                yield i\n                if i != n // i:\n                    yield n // i\n\nclass AdvancedIterators:\n    """Demonstrate advanced iterator patterns and itertools usage."""\n    \n    @staticmethod\n    def sliding_window(data: Iterator[Any], window_size: int) -> Iterator[Tuple[Any, ...]]:\n        """Create sliding windows over data."""\n        iterator = iter(data)\n        window = list(islice(iterator, window_size))\n        \n        if len(window) == window_size:\n            yield tuple(window)\n        \n        for item in iterator:\n            window.pop(0)\n            window.append(item)\n            yield tuple(window)\n    \n    @staticmethod\n    def group_by_key(data: Iterator[Dict[str, Any]], key: str) -> Iterator[Tuple[Any, Iterator[Dict[str, Any]]]]:\n        """Group data by a specific key."""\n        # Sort data by key first (required for groupby)\n        sorted_data = sorted(data, key=lambda x: x.get(key, None))\n        \n        for group_key, group_items in groupby(sorted_data, key=lambda x: x.get(key, None)):\n            yield group_key, group_items\n    \n    @staticmethod\n    def combinations_with_replacement(data: Iterator[Any], r: int) -> Iterator[Tuple[Any, ...]]:\n        """Generate combinations with replacement."""\n        data_list = list(data)\n        for combo in combinations_with_replacement(data_list, r):\n            yield combo\n    \n    @staticmethod\n    def infinite_sequence(start: int = 0, step: int = 1) -> Iterator[int]:\n        """Generate an infinite sequence of numbers."""\n        current = start\n        while True:\n            yield current\n            current += step\n\ndef demonstrate_generators():\n    """Demonstrate the power of generators and iterators."""\n    print("=== Generator and Iterator Demo ===\n")\n    \n    processor = DataProcessor()\n    advanced = AdvancedIterators()\n    \n    # Generate and process large dataset\n    print("Generating large dataset...")\n    dataset_size = 100000\n    raw_data = processor.generate_large_dataset(dataset_size)\n    \n    # Process data through pipeline\n    print("Processing data through pipeline...")\n    filtered_data = processor.filter_even_numbers(raw_data)\n    transformed_data = processor.transform_data(filtered_data)\n    \n    # Process in batches\n    print("Processing in batches...")\n    batch_size = 1000\n    total_processed = 0\n    \n    for batch in processor.batch_process(transformed_data, batch_size):\n        total_processed += len(batch)\n        if total_processed % 10000 == 0:\n            print(f"Processed {total_processed} items...")\n    \n    print(f"Total items processed: {total_processed}")\n    \n    # Demonstrate advanced iterators\n    print("\n=== Advanced Iterator Patterns ===")\n    \n    # Sliding window\n    numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n    print("Sliding window (size 3):")\n    for window in advanced.sliding_window(iter(numbers), 3):\n        print(f"  {window}")\n    \n    # Group by\n    sample_data = [\n        {'category': 'A', 'value': 10},\n        {'category': 'B', 'value': 20},\n        {'category': 'A', 'value': 15},\n        {'category': 'C', 'value': 30}\n    ]\n    \n    print("\nGrouped by category:")\n    for category, items in advanced.group_by_key(iter(sample_data), 'category'):\n        print(f"  {category}: {list(items)}")\n    \n    # Infinite sequence\n    print("\nInfinite sequence (first 10):")\n    infinite_seq = advanced.infinite_sequence(1, 2)\n    for i, num in enumerate(infinite_seq):\n        if i >= 10:\n            break\n        print(f"  {num}")\n\ndef memory_efficiency_demo():\n    """Demonstrate memory efficiency of generators vs lists."""\n    print("\n=== Memory Efficiency Comparison ===")\n    \n    # Using list (memory intensive)\n    print("Creating list with 1 million items...")\n    start_time = time.time()\n    large_list = [i for i in range(1000000)]\n    list_time = time.time() - start_time\n    print(f"List creation time: {list_time:.4f}s")\n    print(f"List memory usage: {len(large_list) * 8} bytes (estimated)")\n    \n    # Using generator (memory efficient)\n    print("\nCreating generator with 1 million items...")\n    start_time = time.time()\n    large_generator = (i for i in range(1000000))\n    gen_time = time.time() - start_time\n    print(f"Generator creation time: {gen_time:.4f}s")\n    print(f"Generator memory usage: minimal")\n    \n    # Process both\n    print("\nProcessing list...")\n    start_time = time.time()\n    list_sum = sum(large_list)\n    list_process_time = time.time() - start_time\n    \n    print("Processing generator...")\n    start_time = time.time()\n    gen_sum = sum(large_generator)\n    gen_process_time = time.time() - start_time\n    \n    print(f"\nResults:")\n    print(f"  List sum: {list_sum}, Processing time: {list_process_time:.4f}s")\n    print(f"  Generator sum: {gen_sum}, Processing time: {gen_process_time:.4f}s")\n    print(f"  Total list time: {list_time + list_process_time:.4f}s")\n    print(f"  Total generator time: {gen_time + gen_process_time:.4f}s")\n\nif __name__ == "__main__":\n    demonstrate_generators()\n    memory_efficiency_demo()

Code Editor

from itertools import chain, islice, tee, groupby, combinations, permutations\nfrom typing import Iterator, List, Dict, Any, Tuple\nimport time\nimport random\n\nclass DataProcessor:\n    """Advanced data processing system using generators and iterators."""\n    \n    def __init__(self):\n        self.processed_count = 0\n        self.memory_usage = 0\n    \n    def generate_large_dataset(self, size: int = 1000000) -> Iterator[int]:\n        """Generate a large dataset without storing it all in memory."""\n        for i in range(size):\n            yield random.randint(1, 1000)\n    \n    def filter_even_numbers(self, data: Iterator[int]) -> Iterator[int]:\n        """Filter even numbers from the dataset."""\n        for item in data:\n            if item % 2 == 0:\n                yield item\n    \n    def transform_data(self, data: Iterator[int]) -> Iterator[Dict[str, Any]]:\n        """Transform numbers into structured data."""\n        for item in data:\n            yield {\n                'value': item,\n                'squared': item ** 2,\n                'is_prime': self._is_prime(item),\n                'factors': list(self._get_factors(item))\n            }\n    \n    def batch_process(self, data: Iterator[Any], batch_size: int = 1000) -> Iterator[List[Any]]:\n        """Process data in batches for memory efficiency."""\n        batch = []\n        for item in data:\n            batch.append(item)\n            if len(batch) >= batch_size:\n                yield batch\n                batch = []\n        if batch:\n            yield batch\n    \n    def parallel_processing(self, data: Iterator[Any], num_streams: int = 4) -> Iterator[Any]:\n        """Split data into multiple streams for parallel processing."""\n        # Create multiple iterators from the same data source\n        iterators = tee(data, num_streams)\n        \n        # Process each stream independently\n        for i, iterator in enumerate(iterators):\n            for item in iterator:\n                yield {\n                    'stream_id': i,\n                    'data': item,\n                    'processed_at': time.time()\n                }\n    \n    def aggregate_results(self, data: Iterator[Dict[str, Any]]) -> Dict[str, Any]:\n        """Aggregate results from processed data."""\n        total_count = 0\n        sum_values = 0\n        prime_count = 0\n        max_value = float('-inf')\n        min_value = float('inf')\n        \n        for item in data:\n            total_count += 1\n            sum_values += item['value']\n            if item['is_prime']:\n                prime_count += 1\n            max_value = max(max_value, item['value'])\n            min_value = min(min_value, item['value'])\n        \n        return {\n            'total_count': total_count,\n            'average_value': sum_values / total_count if total_count > 0 else 0,\n            'prime_count': prime_count,\n            'prime_percentage': (prime_count / total_count * 100) if total_count > 0 else 0,\n            'max_value': max_value,\n            'min_value': min_value\n        }\n    \n    def _is_prime(self, n: int) -> bool:\n        """Check if a number is prime."""\n        if n < 2:\n            return False\n        if n == 2:\n            return True\n        if n % 2 == 0:\n            return False\n        \n        for i in range(3, int(n ** 0.5) + 1, 2):\n            if n % i == 0:\n                return False\n        return True\n    \n    def _get_factors(self, n: int) -> Iterator[int]:\n        """Get all factors of a number."""\n        for i in range(1, int(n ** 0.5) + 1):\n            if n % i == 0:\n                yield i\n                if i != n // i:\n                    yield n // i\n\nclass AdvancedIterators:\n    """Demonstrate advanced iterator patterns and itertools usage."""\n    \n    @staticmethod\n    def sliding_window(data: Iterator[Any], window_size: int) -> Iterator[Tuple[Any, ...]]:\n        """Create sliding windows over data."""\n        iterator = iter(data)\n        window = list(islice(iterator, window_size))\n        \n        if len(window) == window_size:\n            yield tuple(window)\n        \n        for item in iterator:\n            window.pop(0)\n            window.append(item)\n            yield tuple(window)\n    \n    @staticmethod\n    def group_by_key(data: Iterator[Dict[str, Any]], key: str) -> Iterator[Tuple[Any, Iterator[Dict[str, Any]]]]:\n        """Group data by a specific key."""\n        # Sort data by key first (required for groupby)\n        sorted_data = sorted(data, key=lambda x: x.get(key, None))\n        \n        for group_key, group_items in groupby(sorted_data, key=lambda x: x.get(key, None)):\n            yield group_key, group_items\n    \n    @staticmethod\n    def combinations_with_replacement(data: Iterator[Any], r: int) -> Iterator[Tuple[Any, ...]]:\n        """Generate combinations with replacement."""\n        data_list = list(data)\n        for combo in combinations_with_replacement(data_list, r):\n            yield combo\n    \n    @staticmethod\n    def infinite_sequence(start: int = 0, step: int = 1) -> Iterator[int]:\n        """Generate an infinite sequence of numbers."""\n        current = start\n        while True:\n            yield current\n            current += step\n\ndef demonstrate_generators():\n    """Demonstrate the power of generators and iterators."""\n    print("=== Generator and Iterator Demo ===\n")\n    \n    processor = DataProcessor()\n    advanced = AdvancedIterators()\n    \n    # Generate and process large dataset\n    print("Generating large dataset...")\n    dataset_size = 100000\n    raw_data = processor.generate_large_dataset(dataset_size)\n    \n    # Process data through pipeline\n    print("Processing data through pipeline...")\n    filtered_data = processor.filter_even_numbers(raw_data)\n    transformed_data = processor.transform_data(filtered_data)\n    \n    # Process in batches\n    print("Processing in batches...")\n    batch_size = 1000\n    total_processed = 0\n    \n    for batch in processor.batch_process(transformed_data, batch_size):\n        total_processed += len(batch)\n        if total_processed % 10000 == 0:\n            print(f"Processed {total_processed} items...")\n    \n    print(f"Total items processed: {total_processed}")\n    \n    # Demonstrate advanced iterators\n    print("\n=== Advanced Iterator Patterns ===")\n    \n    # Sliding window\n    numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n    print("Sliding window (size 3):")\n    for window in advanced.sliding_window(iter(numbers), 3):\n        print(f"  {window}")\n    \n    # Group by\n    sample_data = [\n        {'category': 'A', 'value': 10},\n        {'category': 'B', 'value': 20},\n        {'category': 'A', 'value': 15},\n        {'category': 'C', 'value': 30}\n    ]\n    \n    print("\nGrouped by category:")\n    for category, items in advanced.group_by_key(iter(sample_data), 'category'):\n        print(f"  {category}: {list(items)}")\n    \n    # Infinite sequence\n    print("\nInfinite sequence (first 10):")\n    infinite_seq = advanced.infinite_sequence(1, 2)\n    for i, num in enumerate(infinite_seq):\n        if i >= 10:\n            break\n        print(f"  {num}")\n\ndef memory_efficiency_demo():\n    """Demonstrate memory efficiency of generators vs lists."""\n    print("\n=== Memory Efficiency Comparison ===")\n    \n    # Using list (memory intensive)\n    print("Creating list with 1 million items...")\n    start_time = time.time()\n    large_list = [i for i in range(1000000)]\n    list_time = time.time() - start_time\n    print(f"List creation time: {list_time:.4f}s")\n    print(f"List memory usage: {len(large_list) * 8} bytes (estimated)")\n    \n    # Using generator (memory efficient)\n    print("\nCreating generator with 1 million items...")\n    start_time = time.time()\n    large_generator = (i for i in range(1000000))\n    gen_time = time.time() - start_time\n    print(f"Generator creation time: {gen_time:.4f}s")\n    print(f"Generator memory usage: minimal")\n    \n    # Process both\n    print("\nProcessing list...")\n    start_time = time.time()\n    list_sum = sum(large_list)\n    list_process_time = time.time() - start_time\n    \n    print("Processing generator...")\n    start_time = time.time()\n    gen_sum = sum(large_generator)\n    gen_process_time = time.time() - start_time\n    \n    print(f"\nResults:")\n    print(f"  List sum: {list_sum}, Processing time: {list_process_time:.4f}s")\n    print(f"  Generator sum: {gen_sum}, Processing time: {gen_process_time:.4f}s")\n    print(f"  Total list time: {list_time + list_process_time:.4f}s")\n    print(f"  Total generator time: {gen_time + gen_process_time:.4f}s")\n\nif __name__ == "__main__":\n    demonstrate_generators()\n    memory_efficiency_demo()

Generators, Iterators, and Memory Efficiency

Key Concepts

Learning Objectives

Master

Develop

Tips

Common Pitfalls

Summary

Exercise

Code Editor

Output