Kafka Architecture: Design Principles for High Throughput and Reliability

1. Design Background

Many internet companies generate massive log data daily, including user behavior, operational metrics, and system monitoring information, which need periodic analysis and statistics.

Traditional log analysis systems provide an offline, scalable solution by aggregating logs into a data warehouse, but real‑time processing often incurs significant latency.

Kafka introduces a novel messaging system with APIs that allow applications to subscribe to log events in real or near‑real time, enabling timely analysis.

2. Persistence Design

Unlike traditional messaging systems, Kafka persists messages to disk to support message replay.

Although disks are often perceived as slow, modern disks can achieve sequential write speeds up to 600 MB/s, far exceeding random I/O performance; Kafka leverages this by using sequential disk writes.

2.1 Sequential Disk I/O

Operating systems optimize sequential reads and writes using techniques such as read-ahead and write-behind . read-ahead pre‑loads large data blocks, while write-behind batches small logical writes into a single large physical write.

The OS also caches disk content in the page cache , allowing all disk I/O to pass through this cache.

Kafka does not use an in‑process cache; instead it relies on the OS page cache , which increases cache capacity and keeps the cache available after a broker restart.

Kafka stores messages by simple append‑only writes without complex indexing structures, which aligns with its sequential consumption model and maximizes throughput.

2.2 Message Expiration Mechanism

Kafka can retain messages for a configurable period (e.g., one week) or until a size threshold (e.g., 2 GB) is reached, rather than deleting them immediately after consumption.

3. High‑Throughput Design

Kafka achieves outstanding throughput; the following points summarize its key design choices.

3.1 Local I/O Optimization

Kafka heavily depends on the OS page cache and uses simple sequential reads and appends, ensuring all I/O is sequential and fully exploiting disk throughput.

3.2 Network I/O Optimization

Kafka reduces small I/O operations and byte copies by batching multiple messages into a single network request.

To avoid excessive copying, Kafka uses the sendfile system call, which transfers data directly from the page cache to the socket, eliminating intermediate user‑space copies.

OS reads data from disk → page cache

Application reads from page cache → user‑space buffer

Application writes buffer → kernel socket buffer

OS sends data from socket buffer → NIC

With sendfile , only the final copy to the NIC remains, reducing the total number of copies and system calls.

3.3 Compression

Kafka provides end‑to‑end compression, storing messages in compressed form on the log and decompressing them only at the consumer side.

4. Stability Design

Stability is crucial; Kafka ensures it through partition replication and comprehensive log backup mechanisms.

4.1 Partitions and Replicas

Topics are divided into partitions; messages are evenly distributed across partitions, enabling load balancing and horizontal scaling. Each partition has multiple replicas, allowing failover when some nodes fail.

4.2 Log Backup and ISR

Each partition has a leader and zero or more followers. The leader handles all reads and writes, while followers replicate the leader’s log. The ISR (in‑sync replica) set contains replicas that are fully caught up; only members of ISR can become the new leader if the current leader fails. ISR changes are persisted in ZooKeeper.

5. References

Kafka official documentation: http://kafka.apache.org