Using PHP Generators to Process Large Data Sets and Prevent Memory Exhaustion

PHP beginners may encounter memory exhaustion errors when handling large data sets or arrays, which can crash applications.

<code>Fatal error: Allowed memory size of xxxxx bytes exhausted (tried to allocate xxxxx bytes) in your_script.php on line xx</code>

PHP memory exhaustion indicates the script tried to allocate more memory than allowed. PHP generators provide a solution by allowing iteration over data without loading everything into memory, which is effective for large data sets.

What is a Generator?

A generator is an efficient way to iterate over a data set because it loads data into memory only when needed, helping avoid memory exhaustion.

How Do Generators Work?

Generators load data on demand using the yield keyword, making them highly efficient for large data sets.

Using Generators: Step‑by‑Step Guide

1. Basic Generator Syntax

To create a generator, define a function with the yield statement; the function returns a generator object that can be iterated.

Example using yield :

<code>function numberGenerator($start = 1, $end = 10) {
    for ($i = $start; $i <= $end; $i++) {
        yield $i;
    }
}

// Using the generator:
foreach (numberGenerator() as $number) {
    echo $number . "<br>";
}
</code>

2. Converting a Regular Function to a Generator

Consider a function that returns an array; converting it to a generator involves adding a yield statement inside the loop.

<code>function getNumbersGenerator($start = 1, $end = 10) {
    for ($i = $start; $i <= $end; $i++) {
        yield $i;
    }
}
</code>

3. Generating Key‑Value Pairs

Generators can also produce associative key‑value pairs:

<code>function keyValueGenerator() {
    yield 'a' => 1;
    yield 'b' => 2;
    yield 'c' => 3;
}
</code>

Sending Data Back to a Generator

Generators can receive input via the send() method.

<code>function receiveDataGenerator() {
    while (true) {
        $data = yield;
        if ($data === 'stop') {
            return; // exit generator
        }
        echo "Received: $data<br>";
    }
}

$generator = receiveDataGenerator();
$generator->send('Hello');
$generator->send('World');
$generator->send('stop');
</code>

Returning a Value from a Generator

A generator can return a final value using the return statement.

<code>function countdownGenerator($start) {
    for ($i = $start; $i >= 0; $i--) {
        yield $i;
    }
    return 'Blastoff!';
}

$gen = countdownGenerator(5);
foreach ($gen as $value) {
    echo "$value<br>";
}
echo $gen->getReturn(); // outputs "Blastoff!"
</code>

Real‑World Use Case

Generators are useful for processing large files or data streams. The following example reads a large file line by line:

<code>function readFileLineByLine($filename) {
    $file = fopen($filename, 'r');
    while (!feof($file)) {
        yield fgets($file);
    }
    fclose($file);
}

foreach (readFileLineByLine('large_file.txt') as $line) {
    echo $line . "<br>";
}
</code>

PHP generators are a powerful tool for memory management and handling large data sets, enabling clearer and more efficient code for developers.