Design and Optimization of Multi‑Data‑Center Redis Synchronization

1. Background

The company needs unit‑based deployment across multiple data‑center units to achieve cross‑data‑center disaster recovery and improve user request latency. Consistent full data across data centers is required for true active‑active operation, which depends on reliable bidirectional data synchronization.

2. Problems with Native Redis

2.1 No native master‑master synchronization

Native Redis only supports master‑slave replication, which cannot provide near‑write capability in remote data centers. Failover requires manual intervention and does not support true active‑active writes.

2.2 Issues when simulating master‑master with sync tools

Data Loopback

Data written locally may be synchronized to another data center and then back to the original, causing duplicate synchronization. The system must identify locally written data versus data received from other centers.

Idempotency

Repeated execution of commands during partial sync must be idempotent.

Write Conflicts

Concurrent writes to the same key in different data centers can lead to inconsistent final values, as illustrated by the example where DC1 writes set a 1 and DC2 writes set a 2 , resulting in divergent values after synchronization.

2.3 Partial Sync (psync) Limitations

Redis uses a circular replication buffer for incremental sync, but its default size (64 MiB) is insufficient for long‑duration cross‑region outages. Increasing the buffer size is not practical.

3. Redis Node Modifications

3.1 RESP Protocol Extension

Each write command destined for replication is assigned an ID and prefixed with #{id}\r\n in the RESP stream. Local client writes keep the original protocol; the master adds the ID before sending to replicas. Remote writes already use the extended protocol.

3.2 Real‑time Write Logging

Local writes are logged sequentially with an index file, while remote writes are not logged to keep log size small. This enables long‑duration partial sync without excessive memory consumption.

3.3 Synchronization Flow Refactor

Full sync triggers a child process to generate an RDB file, which is costly. The new flow first attempts partial sync via the circular buffer; if that fails, it tries to sync using the rlog. Only when both fail does it fall back to full sync.

4. rLog Design

rLog consists of an index file and a log file, both written sequentially to achieve performance comparable to Redis AOF. Files are split when they reach 128 MiB or after one hour with more than 100 k entries. Old log files are retained for one day and then deleted, unless a partial sync is in progress.

4.1 Index File Format

Each entry stores pos (offset in the log file), len (command length), and offset (cumulative offset in the replication buffer).

4.2 Log File Splitting

Splitting is based on size (128 MiB) or time (1 hour with >100 k entries). When a split condition is met, the current index and log files are closed and new ones are created.

4.3 Log File Deletion

Logs older than one day are deleted to avoid unbounded disk usage. During a partial sync, deletion is temporarily disabled.

5. Redis Data Synchronization

5.1 Partial Sync with rLog

The replica first sends psync with its runId and offset. If the master can continue with the circular buffer, it replies +CONTINUE . Otherwise it replies +LPSYNC , prompting the replica to request data via rlog. The master may respond with +LCONTINUE to stream log data, followed by lcommit to update offsets. If rlog cannot satisfy the request, the master falls back to +FULLRESYNC .

5.2 Idempotency Adjustments

Non‑idempotent commands have been modified to ensure safe repeated execution; list‑type commands remain non‑idempotent.

5.3 Data Loopback Handling

Commands originating from remote data centers receive negative IDs and are filtered out, preventing loopback writes.

5.4 Expiration and Eviction

Expiration and eviction are handled locally; such deletions are assigned negative IDs and are not synchronized.

5.5 Data Migration

During cluster expansion, slot migration uses restore and del commands, both assigned negative IDs so they are not cross‑center synchronized.

6. Redis Performance

Benchmarks show that the multi‑active Redis instance with rlog performs on par with native Redis using AOF.

7. Open Issues

7.1 Write Conflicts

Concurrent writes to the same key across data centers remain unresolved; CRDT is a potential future solution.

7.2 List Type Idempotency

List operations are largely non‑idempotent and have not been optimized.

7.3 Expiration/Eviction Consistency

Non‑synchronized eviction can cause temporary inconsistency; the current mitigation is proactive capacity planning.