Redis Hash Internal Encoding: Ziplist vs Hashtable – Performance Analysis and Optimization

During a weekend incident, a sudden rise in latency alerts was traced to slow Redis hash queries. Only a subset of Redis instances showed high CPU usage, and the hot keys were hash structures with hundreds of fields. The issue was mitigated by a business-side degradation plan, but a post‑mortem was needed to understand the root cause.

Case Review

Although the business degradation plan stopped the loss, the Redis service team needed to investigate why only city A, which experienced heavy rain and a three‑fold increase in queued orders, suffered from hot‑key latency while city B did not.

Why did a higher QPS period not cause the same problem?

Why did only city A’s hash objects cause trouble?

The investigation revealed five hot keys in city A, each a hash with >400 elements, accessed mainly by HEXISTS (O(1) complexity). The hash objects in city A used the ziplist internal encoding, while city B’s hashes used hashtable .

Hash Object Internal Encodings

Redis objects have a type (string, list, hash, set, sorted set, etc.) and an encoding (int, ziplist, hashtable, quicklist, etc.). The encoding is chosen dynamically based on size and usage patterns.

Two encodings for hash objects:

ziplist : Compact, memory‑efficient, suitable for small hashes.

hashtable : Faster for large hashes, uses more memory.

Example Redis object definition:

typedef struct redisObject {
// type
unsigned type:4;
// encoding
unsigned encoding:4;
// pointer to underlying data structure
void *ptr;
...
} robj;

Command Execution Logic for Both Encodings

When executing HSET/HMSET , Redis checks the hash object's encoding and may convert it:

Check value size – if > hash_max_ziplist_value (default 64 B) and encoding is ziplist , convert to hashtable .

Check number of fields – if > hash_max_ziplist_entries (default 512) and encoding is ziplist , convert.

Conversion is performed by hashTypeConvert() , which creates a dictionary, copies all field/value pairs, frees the original ziplist, and switches the object's encoding.

Conversion code (simplified):

void hashTypeConvertZiplist(robj *o, int enc) {
serverAssert(o->encoding == OBJ_ENCODING_ZIPLIST);
if (enc == OBJ_ENCODING_HT) {
hashTypeIterator *hi = hashTypeInitIterator(o);
dict *dict = dictCreate(&hashDictType, NULL);
while (hashTypeNext(hi) != C_ERR) {
robj *field = tryObjectEncoding(hashTypeCurrentObject(hi, OBJ_HASH_KEY));
robj *value = tryObjectEncoding(hashTypeCurrentObject(hi, OBJ_HASH_VALUE));
dictAdd(dict, field, value);
}
hashTypeReleaseIterator(hi);
zfree(o->ptr);
o->encoding = OBJ_ENCODING_HT;
o->ptr = dict;
} else {
serverPanic("Unknown hash encoding");
}
}

Time‑Complexity Comparison

Command

ziplist implementation

hashtable implementation

HSET

O(1) –

ziplistPush

twice

O(1) –

dictAdd

HGET

O(N) –

ziplistFind

+

ziplistNext

O(1) –

dictFind

+

dictGetVal

HEXISTS

O(N) –

ziplistFind

O(1) –

dictFind

HDEL

O(N) – locate and remove nodes

O(1) –

dictDelete

HLEN

O(1) –

ziplistLen

/2

O(1) –

dictSize

HGETALL

O(N) – traverse ziplist

O(N) – traverse dict

The key insight: HEXISTS on a ziplist‑encoded hash is O(N), while on a hashtable it is O(1). City A’s hot keys used ziplist (≈400 fields), causing higher latency; City B’s hashes used hashtable, performing better.

Theoretical Test Verification

A performance test compared two hashes (mykey001 – ziplist, mykey002 – hashtable) with 500 fields each (value size 60 B). Both were stressed with 20 k QPS of HEXISTS .

P99 latency: ziplist ≈ 70 µs, hashtable ≈ 60 µs (14 % reduction).

CPU usage: ziplist ≈ 50 %, hashtable ≈ 14 % (72 % reduction).

Memory: ziplist ≈ 4.3 MB, hashtable ≈ 19.6 MB (4.6× larger).

Additional notes on DEBUG OBJECT serialization differences caused by RDB compression were observed.

Optimization Ideas

Server‑side: Adjust hash-max-ziplist-entries or hash-max-ziplist-value to force conversion to hashtable when needed, keeping in mind the memory cost.

Business‑side: For hot keys, either increase the field/value size to trigger hashtable encoding or reduce the number of hash shards so each hash exceeds the entry threshold.

Usage Recommendations:

Avoid hot‑key concentration; distribute load across many keys.

When using list/hash/set/zset, consider the performance impact of the underlying encoding.

Use OBJECT ENCODING to inspect a key’s current encoding.

Conclusion

The case demonstrates that Redis’s dual internal encodings for hash objects lead to significant performance and cost differences. Understanding the encoding choice and tuning related parameters can greatly improve latency and CPU usage, while being aware of the memory trade‑off.