Common Pitfalls and Best Practices of Distributed Caching with Redis and Memcached

In high‑concurrency systems, Redis and Memcached are frequently used as distributed caches to reduce database pressure, but improper cache design can cause various problems.

Memcached server has no built‑in clustering; data distribution is handled by the client (e.g., XMemcachedClientBuilder builder = new XMemcachedClientBuilder(AddrUtil.getAddresses(servers)); ) which by default uses simple modulo hashing, leading to massive key invalidation on node changes. Enabling consistent hashing (Ketama) reduces the impact, though it may cause uneven data distribution. Memcached is multithreaded, stores values up to 1 MB, and evicts expired keys only on the next access.

Redis server supports cluster mode with hash slots (16384 slots) and master‑slave high availability. It processes commands in a single thread, so heavy commands (e.g., KEYS ) can block the server. Data is sharded by slot, and slot reassignment occurs automatically when nodes are added or removed.

Cache structure selection recommends using Memcached for simple key‑value storage (e.g., textual data) and Redis for richer data structures (sorted sets, hashes) to handle pagination, sorting, and ranking, often combining both: Redis holds index information while Memcached stores the detailed payload.

Redis large‑index backfill can be slow when many keys expire; using a message queue to build indexes page by page mitigates the load.

Consistency issues are illustrated with scenarios such as concurrent read/write, master‑slave lag, and cache pollution. Solutions include binlog‑driven cache invalidation, using Canal to listen to DB changes, key versioning (e.g., appending _v2 ), and ensuring single‑threaded consumption of binlog messages.

Hit‑rate problems arise when frequent updates generate many binlog messages; updating the cache directly in the consumer while preserving message order (by grouping tasks by key or ID) helps maintain a high hit rate.

Cache penetration occurs when requests for non‑existent data repeatedly hit the DB; caching empty results with short TTLs and validating input ranges can reduce DB load.

Cache breakdown (thundering herd) is mitigated by adding mutexes or distributed locks around the back‑fill logic to prevent multiple concurrent DB accesses.

Cache avalanche can be avoided through high‑availability configurations, improved sharding strategies, and back‑source rate limiting during massive cache failures.

Hot‑key problems are addressed by creating multiple cache replicas for popular keys and adding a short‑lived local cache layer to spread the load.

Overall, careful design of sharding, invalidation, and concurrency controls is essential for building a robust distributed caching layer.