Challenges and Practices of Distributed Data Systems: Master‑Slave Replication, Partitioning, and High‑Availability Strategies

Distributed Data System Challenges

Distributed systems must balance the three aspects of the CAP theorem: consistency, availability, and partition tolerance, while also handling data synchronization, fault recovery, scalability, and elasticity.

Theoretical Foundations

Consistency : All nodes must see the same data at any point in time.

Availability : The system should continue to serve requests even when some nodes fail.

Partition Tolerance : The system must operate correctly despite network partitions.

Additional concerns include data replication, fault tolerance, and elastic scaling.

Master‑Slave Replication

Why Use Master‑Slave Replication

Master‑slave replication provides read/write separation, data disaster recovery, and load reduction on the primary node. It is especially useful for large‑scale data extraction and for offloading read traffic.

MySQL Replication

The replication process follows these steps:

Write operations on the master are recorded sequentially in the binary log (binlog).

Slaves connect to the master and receive binlog updates.

The master’s binlog‑dump thread pushes changes to each slave.

Each slave’s I/O thread writes the received data to a local relay‑log.

The SQL thread reads the relay‑log and applies the changes to the slave database.

Replication Modes

Synchronous replication : Guarantees that a transaction is committed on both master and slave before returning.

Asynchronous replication (default in MySQL): The master does not wait for slaves, which can lead to lag.

Semi‑synchronous replication : The master waits for at least one slave to acknowledge receipt.

Binlog Formats

MySQL supports three binlog formats:

Statement‑based replication (SBR) : Records each SQL statement. Simple and compact but can cause inconsistencies with nondeterministic functions like NOW() or UUID() .

Row‑based replication (RBR) : Records changes at the row level, ensuring consistency for complex statements but producing larger binlog files.

Mixed mode : Uses SBR by default and switches to RBR when necessary. Since MySQL 5.7 the default binlog_format is MIXED .

Redis Replication

Redis 2.8 introduced a master‑slave replication model that uses a replication buffer on the master. When a slave reconnects, it resumes from the last offset if the data is still in the buffer; otherwise a full sync occurs.

Redis 4.0 added the PSYNC2 protocol to persist replication metadata (run‑id and offset) in the RDB file, enabling partial resynchronization after restarts or master failover.

if(rdbSaveAuxFieldStrStr(rdb,"repl-id",server.replid) == -1) return -1;
if(rdbSaveAuxFieldStrInt(rdb,"repl-offset",server.master_repl_offset) == -1) return -1;

Data Partitioning and Sharding

Partition Algorithms

Modulo : Simple key % N, but suffers from massive data movement when N changes.

Hash : Uses hash functions (e.g., MD5, SHA‑256, CRC16) to map keys to a fixed‑length value, providing uniform distribution but still requiring re‑hashing on scale‑out.

Range : Assigns contiguous key ranges to nodes; easier to expand but can cause hotspot imbalance.

Consistent Hashing : Maps both nodes and keys onto a virtual ring, minimizing data movement during scaling. However, it may produce data skew when the node count is low.

Hash Slots (Redis Cluster) : CRC16(key) mod 16384 determines the slot; each node owns a subset of slots, enabling efficient lookup and limited reshuffling.

Sharding Strategies

Two main deployment models:

Client‑side sharding : Libraries such as ShardingSphere‑JDBC or Alibaba TDDL parse and route SQL in the application.

Proxy‑side sharding : Solutions like ShardingSphere‑Proxy, MyCAT, or JD Elastic DB act as a middleware that handles parsing, routing, and result merging.

Scaling Considerations

Ideal scaling avoids data migration and hotspot creation. A hybrid approach combining modulo for coarse distribution and range for fine‑grained control can achieve this.

Hot‑Key and Big‑Key Management

Large keys (>20 KB) and hot keys can degrade QPS, increase latency, and cause memory imbalance or OOM. Strategies include:

Splitting large collections (lists, sets, sorted sets, hashes) into smaller chunks.

Adding expiration and actively deleting stale hash fields.

Redesigning key schemas to include more granular identifiers, reducing key size.

Introducing random suffixes to hot keys to distribute load across slots.

// Example: delete expired hash fields
allFields.forEach(expireDay -> {
    deleteCache(logPrefix, storeProductionKey, expireDay);
});

private void deleteCache(String logPrefix, StoreProductionKey storeProductionKey, String day) {
    String key = storeProductionKey.generateConfigKey();
    try {
        redisClient.hDel(logPrefix, key, day);
    } catch (Exception e) {
        log.error(logPrefix + "XXX" + key + day, e);
    }
}

Practical Lessons

Real‑world incidents, such as a large DELETE statement causing prolonged MySQL master‑slave lag, illustrate the importance of limiting batch operations, monitoring binlog volume, and avoiding index‑driven full scans.

Truncate operations are high‑risk in production because they cannot be rolled back via binlog and require DROP privileges.

References

1. "Design of Data‑Intensive Application Systems"

2. "Redis Design and Implementation, 2nd Edition"

3. "Redis 5 Design and Source Code Analysis"

4. Articles on consistent hashing and CRC16 slot algorithms.