Kafka Architecture, Performance Optimization, and Production Deployment Guide

Message queues bring decoupling, asynchronous processing, and traffic control to large‑scale systems, enabling scenarios such as e‑commerce flash sales where risk control, inventory locking, order generation, SMS notification, and data updates are orchestrated through a broker.

Kafka’s core concepts include producers that write to a cluster, consumers that pull data, topics that are logical streams, partitions that provide parallelism, brokers (Kafka servers) that host partitions, and replication with leader and follower replicas managed via the ISR (in‑sync replica) list.

High‑performance data flow relies on sequential disk writes, OS page cache, and zero‑copy transfer (sendfile). A producer writes to the OS cache, which is flushed to disk; the consumer reads from the cache or disk, the data is copied to the application, then sent through the socket cache to the network interface.

Capacity planning for a workload of 1 billion daily requests (≈5.5 × 10⁴ QPS at peak) suggests a five‑node physical cluster. Each node with 11 × 7 TB SAS disks (≈77 TB total) provides enough storage for the estimated 276 TB of retained data. Memory sizing targets 64 GB (128 GB ideal) to keep hot log segments in cache, and 16–32 CPU cores to handle the >100 k thread count of Kafka processes.

Resource evaluation includes:

Disk: 5 nodes × 11 × 7 TB ≈ 385 TB raw capacity.

Memory: 250 GB of hot log data divided across partitions, plus 10 GB JVM heap, totaling ~60 GB per node.

CPU: Minimum 16 cores per node; 32 cores preferred for headroom.

Network: 1 GbE (≈700 Mbps usable) or 10 GbE for peak traffic of ~1 GB/s per node.

Key producer configuration parameters for throughput include acks (0, 1, -1), batch.size , linger.ms , compression.type , buffer.memory , retries , and max.in.flight.requests.per.connection . Adjusting these balances latency, durability, and CPU usage.

A custom partitioner can route specific keys to a dedicated hot‑data partition. Example Java code:

public class HotDataPartitioner implements Partitioner {
    private Random random;
    @Override
    public void configure(Map
configs) { random = new Random(); }
    @Override
    public int partition(String topic, Object keyObj, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        String key = (String) keyObj;
        List
partitionInfoList = cluster.availablePartitionsForTopic(topic);
        int partitionCount = partitionInfoList.size();
        int hotDataPartition = partitionCount - 1;
        return !key.contains("hot_data") ? random.nextInt(partitionCount - 1) : hotDataPartition;
    }
}

Configure it with props.put("partitioner.class", "com.zhss.HotDataPartitioner");

Consumer groups use a coordinator (selected via the __consumer_offsets topic) to manage heartbeats, detect failures, and trigger rebalance. Rebalance strategies include range , round‑robin , and sticky , each distributing partitions differently among consumers.

Operational tools such as KafkaManager and the kafka-reassign-partitions.sh script simplify topic creation, partition reassignment, and replica factor changes. Example commands:

kafka-topics.sh --create --zookeeper hadoop1:2181,hadoop2:2181,hadoop3:2181 \
    --replication-factor 1 --partitions 1 --topic test6
kafka-reassign-partitions.sh --zookeeper hadoop1:2181,hadoop2:2181,hadoop3:2181 \
    --reassignment-json-file test.json --execute

Kafka stores consumer offsets in the internal __consumer_offsets topic (default 50 partitions) to avoid Zookeeper bottlenecks, enabling high‑throughput offset commits.

Delayed operations (e.g., ack timeouts, follower fetch waits) are managed by a multi‑level time‑wheel mechanism, providing O(1) insertion and removal of timers and efficiently handling millions of scheduled tasks.