Redis Core Architecture, Data Types, Persistence, High Availability, and Performance Optimization

Redis is an in‑memory database that provides extremely fast read/write operations, but its data must be persisted to survive crashes. The article walks through the complete architecture, from the client layer to the event‑driven network core, command parsing, memory allocation, persistence, high‑availability, and monitoring.

Core Architecture

The server is represented by the redisServer structure (defined in server.h ) which holds fields such as pid , configfile , redisDb *db , aeEventLoop *el , networking ports, client lists, and replication flags.

struct redisServer {
    pid_t pid;               /* main process pid */
    pthread_t main_thread_id;
    char *configfile;        /* path to redis.conf */
    redisDb *db;             /* array of databases */
    int dbnum;               /* number of DBs */
    dict *commands;          /* command table */
    aeEventLoop *el;         /* event loop */
    int sentinel_mode;       /* sentinel flag */
    int port;                /* TCP listening port */
    list *clients;           /* connected clients */
    client *current_client;  /* client executing command */
    /* ... other fields omitted for brevity ... */
};

Data Types and Underlying Structures

Redis supports many data types, each with a specific internal representation:

String – implemented with the SDS (Simple Dynamic String) structure that stores length, allocation size, and a pointer to the buffer, enabling O(1) length queries.

List – historically used linkedlist and ziplist ; later replaced by quicklist (a linked list of ziplist nodes) and finally by listpack for compact storage.

Set – stored as a hash table; when all members are small integers it can use an intset array for memory efficiency.

Hash – implemented either as a hash table ( dict ) or as a listpack when the number of fields is small.

Sorted Set – uses a skiplist for fast range queries combined with a hash table for O(1) member lookup; for small sets it falls back to listpack .

Bitmap – a bit‑level view on a String, allowing 8 bits per byte and massive space savings.

HyperLogLog – a probabilistic structure that estimates cardinality using 12 KB of memory; its header is defined as: struct hllhdr { char magic[4]; uint8_t encoding; uint8_t notused[3]; uint8_t card[8]; uint8_t registers[]; };

Geo – built on top of Sorted Sets using the GeoHash algorithm to encode latitude/longitude into a single sortable integer.

Stream – a log‑structured data type that stores messages in a Radix Tree and listpack , providing consumer groups similar to Kafka.

Persistence Mechanisms

Redis offers two main persistence options:

RDB snapshots – the server forks a child process (using copy‑on‑write) that writes the in‑memory data to a compact binary file. The forked child does not block the main thread, allowing continued client service.

AOF (Append‑Only File) – logs every write command. The file format looks like: *3 $3 SET $3 key $5 value AOF can be rewritten to a smaller file by creating a new child process that re‑executes the current dataset as a series of minimal commands.

Hybrid persistence combines periodic RDB snapshots with incremental AOF logs, reducing the frequency of heavy fork operations.

High Availability

Replication copies data from a master to one or more slave instances. In case of master failure, Sentinel monitors the topology, performs leader election, and promotes a slave to master, handling automatic failover.

Redis Cluster shards the keyspace into 16384 slots, each node owning a subset of slots. It provides horizontal scaling, automatic failover, and uses the Gossip protocol for node discovery.

Performance Monitoring and Optimization

Baseline latency can be measured with redis-cli --intrinsic-latency 100 . Slow‑log and latency-monitor help identify commands that exceed acceptable latency.

Common performance pitfalls and solutions include:

Network RTT – use pipelining to batch commands and reduce round‑trips.

Slow commands (O(N)) – replace with O(1) or O(log N) alternatives, or use incremental iteration commands like SCAN , SSCAN , HSCAN , ZSCAN .

Transparent HugePages – disable to avoid copying 2 MB pages during fork.

Swap usage – monitor /proc/ /smaps for swapped pages; avoid swapping by provisioning enough RAM or isolating Redis on its own machine.

AOF fsync settings – choose no or everysec for typical cache workloads to reduce disk I/O.

Fork overhead for RDB – keep dataset size moderate (2–4 GB) or use replicas for snapshot generation.

Expiration handling – spread expirations with a small random offset to avoid massive synchronous cleanup.

Bigkeys – detect with redis-cli --bigkeys and split or delete using UNLINK to avoid blocking.

Multithreaded I/O

Since Redis 6.0, network I/O can be processed by multiple threads while command execution remains single‑threaded, improving throughput for high‑concurrency workloads.

Overall, understanding Redis's internal data structures, persistence strategies, replication mechanisms, and performance‑tuning knobs enables developers to build robust, high‑performance systems that fully leverage Redis's capabilities.