Cache Design and Optimization Strategies in High‑Concurrency Distributed Systems

Introduction

In high‑concurrency distributed systems, caching is indispensable for accelerating reads and shielding backend services from overwhelming request loads; therefore, designing an effective cache layer is crucial.

Cache Benefits and Costs

Benefits include accelerated read/write performance (e.g., Redis can handle tens of thousands of QPS while MySQL handles only a few thousand) and reduced backend load by offloading expensive computations.

Costs involve potential data inconsistency due to expiration windows, increased code‑maintenance effort, and operational overhead such as ensuring high availability of cache clusters.

Cache Update Strategies

Three primary eviction algorithms are discussed:

LRU – evicts the least recently used items.

LFU – evicts items with the lowest access frequency.

FIFO – evicts items in first‑in‑first‑out order.

These are suitable when memory is limited and data changes rarely.

Other strategies include:

Timeout eviction using commands like Redis expire , which balances consistency and simplicity.

Active update, where the cache is refreshed immediately after the data source changes, offering the highest consistency at the cost of higher development effort.

Best practices suggest combining low‑consistency strategies (LRU/FIFO) with timeout eviction for low‑consistency workloads, and combining timeout eviction with active update for high‑consistency scenarios.

Penetration Optimization

Cache penetration occurs when queries for non‑existent data miss both cache and storage, potentially overwhelming the backend. Two mitigation approaches are presented:

Caching empty objects with a short TTL after a miss.

Using a Bloom filter to pre‑check existence before accessing the storage layer.

Both can be combined for effective protection.

Bottom‑Hole Optimization

The “bottom‑hole” problem arises when distributed hash‑based key placement causes excessive network hops for batch operations (e.g., mget ). Solutions include serial, parallel, and hash‑tag based batch fetching, each reducing network round‑trips and improving latency.

Avalanche Optimization

Cache avalanche happens when the cache becomes unavailable, causing a sudden surge of requests to the storage layer. Prevention measures include ensuring cache high availability (master‑slave, Sentinel), employing rate‑limiting and circuit‑breaker patterns (e.g., Netflix Hystrix), and isolating project resources.

Hot‑Key Rebuild Optimization

When hot keys expire, simultaneous cache rebuilds can overload the backend. Mitigation techniques include:

Mutex locking so only one thread rebuilds while others wait or use stale data.

Never‑expire keys with periodic background refreshes.

Backend rate‑limiting to throttle rebuild attempts.

Combining these strategies helps avoid snowballing load spikes.

Conclusion

The author shares practical insights on cache design, emphasizing the trade‑offs between performance gains and operational complexity, and encourages readers to apply the discussed patterns to build robust, high‑performance backend systems.