Understanding Redis AOF Rewrite Mechanism and Its Implementation

What Is AOF Rewrite

AOF (Append‑Only File) is one of Redis's persistence methods. It records every write command to an AOF log, but when a key is modified repeatedly the log contains many redundant commands, causing the file to grow and recovery to become slower. Redis solves this by providing an AOF rewrite feature.

How the AOF Rewrite Buffer Is Defined

Redis does not simply use malloc and realloc for the rewrite buffer. Instead it defines a aofrwblock structure that contains a fixed 10 MB character array, allocating one block at a time and linking them via the aof_rewrite_buf_blocks list.

typedef struct aofrwblock {
    unsigned long used, free;
    char buf[AOF_RW_BUF_BLOCK_SIZE]; // 10MB
} aofrwblock;

This design avoids the overhead of realloc which could trigger Linux's huge‑page allocation and waste memory when a client write touches data that is being persisted.

AOF Rewrite Process

The rewrite runs in a background child process created with fork() . The parent continues to serve clients while buffering new write commands in aofrwblock structures. When the child finishes generating a temporary AOF file, the parent merges the buffered data and atomically replaces the old AOF.

Parent Process Steps

If an AOF rewrite or RDB child is already running, the rewrite is aborted.

Three pipes are created for communication: parent‑>child data, child‑>parent ACK, and parent‑>child ACK.

The write end of the data pipe is set to non‑blocking.

Read events are registered on the ACK pipe.

File descriptors are stored in the server structure.

The parent records the start time for statistics.

fork() is called to create the child.

Child Process Steps

Close listening sockets to avoid new client connections.

Set the process title to "redis‑aof‑rewrite".

Create a temporary file name for the new AOF.

Iterate over every database and every key, writing INSERT‑style commands to the temporary file.

Pipe Communication Mechanism

Three pipes are used:

Parent → Child data (writes from parent to child).

Child → Parent ACK (child notifies parent that it wants the parent to stop sending diffs).

Parent → Child ACK (parent acknowledges the child's request).

During rewriting, the child repeatedly calls aofReadDiffFromParent() to read buffered diffs, and the parent writes new diffs via aofChildWriteDiffData() . When the child finishes, it sends a "!" byte; the parent receives it, stops sending further diffs, and replies with its own "!".

ssize_t aofReadDiffFromParent(void) {
    char buf[65536]; // typical Linux pipe buffer size
    ssize_t nread, total = 0;
    while ((nread = read(server.aof_pipe_read_data_from_parent, buf, sizeof(buf))) > 0) {
        server.aof_child_diff = sdscatlen(server.aof_child_diff, buf, nread);
        total += nread;
    }
    return total;
}

When AOF Rewrite Is Triggered

Manual trigger via the BGREWRITEAOF command.

Automatic trigger when the AOF size grows beyond a configured percentage or absolute size.

Scheduled trigger when the server's cron loop detects the conditions.

Drawbacks of AOF Rewrite

Memory Overhead

During rewriting both the original AOF buffer and the rewrite buffer hold similar data, potentially exhausting the configured maxmemory limit.

CPU Overhead

The parent must copy data into the rewrite buffer and drive the event loop to send it to the child, consuming CPU cycles proportional to the amount of pending data.

Disk I/O Overhead

Both the parent and child write similar data to disk, effectively doubling I/O for the same logical operation.

Fork Cost

Although the fork uses copy‑on‑write, the initial duplication of the page table can block the server for a noticeable time, especially on large instances.

Reference Materials

GeekTime column "Redis Source Code Analysis and Practice" (2021).

GeekTime column "Redis Core Technology and Practice" (2020).

Redis 7.0 Multi‑Part AOF Design and Implementation (2022).

Redis 5.0.8 source code on GitHub.