Designing a Multi‑Datacenter Redis Master‑Slave Architecture with Automatic Failover

Background

The author is planning a Redis deployment that spans multiple data‑centers with a primary‑replica architecture, aiming to store critical and sensitive data in Redis and synchronize it to a database via an asynchronous message queue.

Solution Overview

1. Data‑center and Master Definition

Three data‑centers are assumed: JF-A, JF-B, and JF-C. Write operations occur only in JF-A, so the Redis master is placed there.

To prepare for disaster recovery, a Redis master backup is deployed alongside the master in JF-A, with automated failover handling.

In case JF-A loses network connectivity, an additional Redis master backup is deployed in JF-B, and operational procedures control master‑backup relationships across all sites.

2. Deployment per Data‑center

JF‑A master: JF-A-Master JF‑A backup (slave of master): JF-A-M-Backup (slaveof JF-A-Master)

JF‑B backup (slave of master): JF-B-M-Backup (slaveof JF-A-Master)

JF‑A slave 1: JF-A-S-1 (slaveof JF-A-Master)

JF‑A slave 2: JF-A-S-2 (slaveof JF-A-S-1) – same machine, different instance

JF‑B slave 1: JF-B-S-1 (slaveof JF-A-Master)

JF‑B slave 2: JF-B-S-2 (slaveof JF-B-S-1) – same machine, different instance

JF‑C slave 1: JF-C-S-1 (slaveof JF-A-Master)

JF‑C slave 2: JF-C-S-2 (slaveof JF-C-S-1) – same machine, different instance

Master Configuration Check

The diagram shows the backup masters and slaves (slave0~4) across the three data‑centers, with additional slave instances attached in a tree‑like fashion.

Master‑Slave Synchronization Scheme

Redis uses a Pull&Push model for replication:

When a new slave is added, it initiates a Pull by sending a SYNC command to the master to fetch the initial dataset.

Upon data changes, the master iterates over connected_slaves and pushes updates to each slave.

Slaves maintain a heartbeat (configured via repl-ping-slave-period, default 10 s) to keep the connection alive.

Redis Master‑Slave Implementation Details (Excerpt)

1. Slave connects and sends SYNC

int syncWithMaster(void) {
  // login to master if authentication is required
  if(server.masterauth) {
    syncWrite(fd, "AUTH xxx
", strlen(server.masterauth)+7, 5);
    // ...
  }
  // send SYNC command
  syncWrite(fd, "SYNC 
", 7, 5);
  // open temporary RDB file for incoming data
  int dfd = open(tmpfile, O_CREAT|O_WRONLY|O_EXCL, 0644);
  // create read event for the RDB payload
  aeCreateFileEvent(server.el, fd, AE_READABLE, readSyncBulkPayload, NULL);
  return REDIS_OK;
}

2. Master creates RDB file

void syncCommand(redisClient *c) {
  // if a background save is already in progress, wait for it
  if(server.bgsavechildpid != -1) {
    // ... wait logic ...
  } else {
    // otherwise trigger RDB generation for the SYNC
    rdbSaveBackground(server.dbfilename);
  }
}

After rdbSaveBackground() finishes, updateSlavesWaitingBgsave() is called to send the RDB file to all waiting slaves.

void updateSlavesWaitingBgsave(int bgsaveerr) {
  listRewind(server.slaves, &li);
  while((ln = listNext(&li))) {
    slave->repldbfd = open(server.dbfilename, O_RDONLY);
    // create writable event to push the bulk data
    aeCreateFileEvent(server.el, slave->fd, AE_WRITABLE, sendBulkToSlave, slave);
  }
}

Each slave receives the bulk RDB payload, clears its current dataset, and loads the new data.

3. Incremental Synchronization

void call(redisClient *c, struct redisCommand *cmd) {
  if((dirty || cmd->flags & REDIS_CMD_FORCE_REPLICATION) && listLength(server.slaves)) {
    replicationFeedSlaves(server.slaves, c->db->id, c->argv, c->argc);
  }
  // ... other command handling ...
}

The call() function is invoked for each client command; if the command modifies data ( dirty) and there are slaves, replicationFeedSlaves() streams the command to all replicas, achieving real‑time incremental sync.