RocksDB Memory Usage Analysis and Optimization: Troubleshooting Excessive Memory Consumption in Production

This article provides an in-depth analysis of RocksDB memory usage issues encountered in production environments. RocksDB, a high-performance key-value storage engine, is widely used in scenarios requiring high throughput and low latency. However, developers often observe memory usage exceeding预设 thresholds, threatening system stability.

The author presents a real case study from vivo's production environment where two RocksDB instances (with block-cache-size configured at 4GB and 8GB respectively) consumed 59GB of memory on a 32C64G server, approaching the physical memory limit. The analysis process involved multiple steps: initial suspicion of RocksDB memory leaks, investigation using tools like jeprof, and ultimately identifying the root cause in glibc's ptmalloc memory allocator.

Key findings include: RocksDB's block-cache-usage and estimate-table-readers-mem accounted for approximately 15.9GB, while the remaining ~4GB was held by glibc's ptmalloc as unreturned free memory. The solution involved replacing glibc ptmalloc with jemalloc, resulting in 25% memory reduction (from 95%+ to 80%+) and improved IO/CPU utilization.

The article demonstrates effective memory analysis methodologies using Linux tools (free, top, vmstat, pidstat) and profiling tools (jemalloc's jeprof, malloc_info), providing valuable insights for developers facing similar memory management challenges.