Cache Warm-up Strategies and Lessons Learned from a Production Incident

Cache Warm-up Strategies and Lessons Learned

What happens if a cache is not pre‑warmed? The system’s interface performance degrades, database pressure increases, and the author was scolded for a two‑day post‑mortem report during a review meeting.

Disastrous Deployment Moment

Early in the author's career, they needed to display inventory status for virtual goods. The calculation was complex and took over 500 ms, which was unacceptable for the product page.

The solution was to cache the inventory status: read from cache if present, otherwise compute, store in cache, and set an expiration. Writes did not synchronously update the cache, but the product tolerated a few minutes of inconsistency because redundant stock could absorb overselling.

The author dove deep into Redis internals, commands, and performance characteristics, but missed a critical design flaw during deployment.

When the feature went live, alarms started firing due to massive latency spikes on the product page. The team asked the author to roll back, but no rollback or degradation switch had been implemented, leading to panic.

How to Pre‑warm a Cache

Gray Release (Gradual Traffic Ramp‑up)

Although not a true warm‑up method, gradually releasing traffic (e.g., 1 % initially) avoids a cache avalanche by limiting load while the cache gradually fills.

Scanning the Database to Populate Cache

For cache keys based on products or users, a full‑table scan can pre‑load data into the cache. This approach requires significant development effort: writing scanning jobs, managing thread pools, and applying rate limiting.

Using a Data Platform to Refresh Cache

A more efficient solution is to leverage a data platform: export offline DB data to Hive, sync Hive to Kafka, then consume Kafka to write into the cache. This ETL‑driven pipeline allows complex SQL transformations, immediate deployment, and built‑in concurrency control, but it requires company‑wide support for multiple storage systems (MySQL, Kafka, Hive, etc.).

Other Scenarios Requiring Cache Warm‑up

If Redis Crashes, What Happens to Data?

When Redis fails, all cached data disappears, causing a sudden surge of requests to the database. Warm‑up tasks should be run not only before launch but also after a cache outage to restore data quickly.

Cold‑start with Massive Data (e.g., Promotional Events)

During events like a Chinese New Year red‑packet giveaway, many inactive users may suddenly become active, leading to massive cache misses. Pre‑warming via ETL (loading data into Kafka then into cache) mitigates this risk.

Conclusion

Always pre‑warm caches; otherwise, online interfaces and databases cannot sustain the load.

Use gray‑release traffic, full‑table scans, or ETL data sync to warm caches.

Prepare for Redis failures and cold‑start scenarios by having automated warm‑up processes.