KeyDB Multithreaded Architecture, Connection Management, Fastlock, and Active‑Replica Mechanism

Today we introduce KeyDB, a branch forked from Redis. Redis is a single‑threaded in‑memory KV store, while KeyDB, while 100% compatible with the Redis API, transforms it into a multithreaded system.

Public technical details are scarce; this article is based on source‑code analysis.

1 Multithreaded Architecture

Thread Model

KeyDB splits the original Redis main thread into a main thread and worker threads. Each worker thread is an I/O thread that listens on ports, accepts connections, reads data and parses the protocol.

KeyDB uses the SO_REUSEPORT feature, allowing multiple threads to bind to the same port.

Each worker thread is bound to a CPU core and uses SO_INCOMING_CPU to specify which CPU receives the data.

After protocol parsing, each thread operates on the in‑memory data, guarded by a single global lock.

The main thread is also a worker thread (index 0 in the worker array) and additionally performs tasks that only the main thread may execute.

The main thread’s responsibilities are implemented in serverCron , including:

Statistics processing

Client connection management

Database resize and reshard

AOF handling

Replication master‑slave synchronization

Cluster‑mode tasks

Connection Management

In Redis all connection management is done in a single thread. In KeyDB each worker thread manages its own set of connections; a connection is inserted into the list belonging to the thread that created it and must be created, processed and destroyed by the same thread. A new field int iel; is added to indicate the owning thread.

KeyDB maintains three key data structures for connection management:

clients_pending_write : per‑thread list of connections awaiting synchronous writes.

clients_pending_asyncwrite : per‑thread list of connections awaiting asynchronous writes.

clients_to_close : global list of connections that need to be closed asynchronously.

Separate synchronous and asynchronous queues are needed because some Redis APIs (e.g., pub/sub) involve sending data from a thread different from the one that owns the client.

The synchronous sending logic is straightforward and stays within the same thread; the diagram below illustrates it.

Asynchronous sending involves two threads communicating via a pipe:

int fdCmdWrite; // write pipe
int fdCmdRead;  // read pipe

When a local thread needs to send data asynchronously, it checks whether the client belongs to the local thread; if not, it obtains the client’s owning thread ID and posts an AE_ASYNC_OP::CreateFileEvent operation to that thread, which adds a write event to its event loop.

Redis sometimes closes client connections from a thread other than the one that owns the connection, so a global asynchronous‑close list is maintained.

Lock Mechanism

KeyDB implements a spin‑lock‑like mechanism called fastlock.

Key data structures of fastlock are:

struct ticket {
    uint16_t m_active; // unlock +1
    uint16_t m_avail;  // lock +1
};

struct fastlock {
    volatile struct ticket m_ticket;
    volatile int m_pidOwner; // thread ID that currently holds the lock
    volatile int m_depth;    // recursion depth for the owning thread
};

Atomic operations __atomic_load_2 , __atomic_fetch_add , and __atomic_compare_exchange are used to compare m_active == m_avail and decide whether the lock can be acquired.

fastlock provides two acquisition methods:

try_lock : returns immediately on failure.

lock : busy‑waits, and after 1 024 × 1 024 iterations yields the CPU with sched_yield .

KeyDB combines try_lock with event handling to avoid busy‑waiting; each client has a dedicated lock that is attempted before reading client data, and if acquisition fails the operation is retried in the next epoll_wait cycle.

2 Active‑Replica

KeyDB implements an active‑replica mechanism where each replica can be writable (non‑read‑only) and replicas synchronize data with each other. Main features include:

Each replica carries a UUID to break circular replication.

A new rreplay API packages incremental commands with the local UUID.

Keys and values are stamped with a timestamp version; if a local key has a newer timestamp than an incoming one, the write is rejected. The timestamp is generated by left‑shifting the current time by 20 bits and adding a 44‑bit auto‑incremented counter.

3 Conclusion

ApsaraDB for Redis is a stable, high‑performance, elastically scalable cloud database service built on the Feitian distributed system and all‑SSD storage, offering both primary‑standby and cluster high‑availability architectures, as well as disaster recovery, online scaling, and performance‑optimization solutions.