How KeyDB Transforms Redis with Multi‑Threading: Architecture & Locks

KeyDB is a fork of Redis that retains 100 % Redis API compatibility while converting the original single‑threaded KV memory store into a multi‑threaded system.

Thread Model

KeyDB splits Redis’s original main thread into a main thread and multiple worker threads. Each worker thread is an I/O thread that listens on the port, accepts connections, reads data and parses the protocol.

KeyDB uses the SO_REUSEPORT feature, allowing multiple threads to bind to the same listening port. Each worker thread is bound to a CPU core and uses the SO_INCOMING_CPU feature to designate which CPU receives data.

After protocol parsing, each thread operates on in‑memory data protected by a global lock to control concurrent access.

The main thread is also a worker thread (index 0 in the worker array) and performs tasks that only the main thread can handle, such as serverCron.

Statistics processing

Client connection management

Database resize and reshard

AOF handling

Replication master‑slave synchronization

Cluster‑mode tasks

Connection Management

In Redis all connection management is done in a single thread. In KeyDB each worker thread manages its own set of connections; connections are inserted into the thread‑local list and must be created, used, and destroyed within the same thread.

int iel; /* the event loop index we're registered with */

KeyDB maintains three key data structures for connection management:

clients_pending_write

: per‑thread list of connections pending synchronous writes.

clients_pending_asyncwrite

: per‑thread list of connections pending asynchronous writes.

clients_to_close

: global list of connections that need asynchronous closing.

Separate synchronous and asynchronous queues handle cases where the thread that generated a command differs from the thread that must send the response (e.g., Pub/Sub). The following diagram illustrates synchronous client writes.

When a thread needs to send data asynchronously, it checks whether the client belongs to the local thread; if not, it obtains the client’s owning thread ID and enqueues a write event via

AE_ASYNC_OP::CreateFileEvent

.

Redis sometimes closes client connections outside the owning thread, so KeyDB also maintains a global asynchronous close list.

Lock Mechanism

KeyDB implements a spin‑lock‑like mechanism called fastlock . Its main data structures are:

struct ticket {
    uint16_t m_active;  // unlock +1
    uint16_t m_avail;   // lock +1
};

struct fastlock {
    volatile struct ticket m_ticket;
    volatile int m_pidOwner; // thread ID holding the lock
    volatile int m_depth;     // recursion depth
};

Atomic operations such as

__atomic_load_2

,

__atomic_fetch_add

, and

__atomic_compare_exchange

are used to compare

m_active

and

m_avail

to determine lock acquisition.

Fastlock provides two lock acquisition methods:

try_lock

: returns immediately on failure.

lock

: busy‑waits, yielding the CPU after 1 048 576 attempts via

sched_yield

.

KeyDB combines

try_lock

with the event loop to avoid busy‑waiting; each client has a dedicated lock, and if locking fails, the operation is deferred to the next

epoll_wait

cycle.

Active‑Replica

KeyDB supports an active‑replica mode where replicas can be writable. Replicas synchronize data with each other and include features such as:

Each replica has a UUID to prevent circular replication.

A new

rreplay

API packages incremental commands with the local UUID.

Keys and values carry a timestamp version; conflicts are resolved by comparing timestamps, using a 64‑bit value where the high 20 bits are the current timestamp and the low 44 bits are an increment.

Project Repository

https://github.com/JohnSully/KeyDB