Strategies for Splitting Large Keys and Values in Redis

In many business scenarios Redis encounters large keys, oversized values, and massive numbers of keys, which can significantly increase memory usage and slow down the single‑threaded server. The article first outlines four typical problem cases: a single key with a huge value, data structures (hash, set, zset, list) holding millions of elements, clusters storing billions of keys, and large Bitmaps or Bloom filters.

1. Splitting a single large value – If the object must be stored and retrieved as a whole, break it into several key‑value pairs and use multiGet to fetch them in parallel, spreading I/O load across multiple Redis instances. If only part of the object is needed, store it as a hash where each field represents a property, and use hget , hmget for reads and hset , hmset for updates.

2. Reducing elements in a value – For structures with many elements, partition them into buckets. For example, fix a bucket count (e.g., 10,000), compute bucket = hash(field) % 10000 , and store each element in a separate hash key like newHashKey = originalHashKey + ":" + bucket . This distributes load but may require additional metadata for ordered operations such as lpop .

3. Handling billions of keys – Excessive keys increase memory due to per‑key overhead and slot‑to‑key mappings in cluster mode. Consolidate related keys into a single hash: the original keys become fields of a new hash key (e.g., user.zhangsan with fields id , age , country ). For unrelated keys, pre‑allocate a fixed number of buckets (e.g., 2 million) and map each key to a bucket using hash(key) % bucketCount , then store the actual data as a field within that bucket’s hash.

4. Splitting large Bitmaps or Bloom filters – When a Bitmap or Bloom filter occupies hundreds of megabytes, divide it into many smaller pieces (e.g., split a 512 MB Bitmap into 1024 pieces of 512 KB each). Assign each piece to a distinct Redis key based on a hash of the original key, ensuring a request touches only one key. This does not increase the Bloom filter false‑positive rate because the ratio n/m remains constant if the distribution is uniform. The article also recommends using the built‑in setBits / getBits API (Redis ≥2.3.4) and keeping each Bloom filter under 512 KB with about 13 hash functions.

These techniques aim to reduce memory consumption, balance I/O load, and maintain Redis performance in high‑scale environments.