Kafka Server Network Framework: Reactor Pattern and Multi‑Threaded Architecture

1. Reactor Pattern

Kafka's network layer adopts the Reactor pattern, an event‑driven model built on Java NIO. The typical single‑threaded Java NIO model registers a ServerSocketChannel with a Selector for OP_ACCEPT events, where the ServerSocketChannel listens for incoming connections.

Process:

First, a ServerSocketChannel object is created and OP_ACCEPT is registered on a Selector; the ServerSocketChannel listens on a specific port for connection requests.

When a client initiates a connection, the Selector detects the OP_ACCEPT event and triggers an Acceptor to handle it.

Upon receiving the client’s socket request, the Acceptor creates a corresponding SocketChannel, sets it to non‑blocking mode, and registers I/O events such as OP_READ and OP_WRITE on the Selector, completing the socket connection.

When the client sends a request through this socket, the Selector detects an OP_READ event and invokes the appropriate handler (ReaderHandler) to process the request.

All event‑handling logic runs in the same thread, which works for a small number of concurrent connections and low data volume but can cause thread blockage when request processing is complex, leading to timeouts.

To avoid this, Kafka introduces multithreading: network read/write logic and business processing are split into separate thread pools, allowing high‑concurrency handling.

The Acceptor can run in its own thread or be backed by a single‑threaded ExecutorService, which automatically replaces a failed thread, preventing the whole server from stopping. ReaderThreadPool threads register OP_READ on the Selector, each handling multiple sockets; after reading a request, they place it into a shared MessageQueue.

HandlerThreadPool threads take requests from MessageQueue, execute business logic, generate responses, and register OP_WRITE on the Selector to send data back to clients. This design ensures that even if some business threads block, others continue processing, avoiding full server blockage.

The length of MessageQueue is critical when read and processing speeds differ.

Typically, the Acceptor occupies its own Selector; when it detects OP_ACCEPT, it creates a SocketChannel and assigns it to a selector (using round‑robin or least‑loaded strategies) that will monitor the channel’s I/O events.

2. SocketServer – Kafka

Kafka’s network layer is multi‑threaded with multiple selectors. The core class is SocketServer, which contains an Acceptor for accepting new connections. Each Acceptor is associated with several Processor threads, each with its own Selector for reading requests and writing responses. Each Acceptor also works with multiple Handler threads that process requests and return responses to the Processor threads via a RequestChannel.

Source: https://happyer.github.io/2017/06/12/2017-06-12-kafka-server-network-framwork/

Copyright Notice: Content sourced from the internet; copyright belongs to the original author. We credit the author and source where possible. If any infringement is identified, please inform us for immediate removal.