In-Depth Overview of Apache Kafka Architecture and Core Concepts

Apache Kafka is a distributed streaming platform that functions as a publish/subscribe‑based message queue, offering three key characteristics: publish‑subscribe messaging, durable storage with fault tolerance, and real‑time processing of streaming records.

It supports two messaging models: point‑to‑point (queue) where each message is consumed by a single consumer, and publish/subscribe (topic) where messages are delivered to all subscribed consumers.

Kafka is suitable for building real‑time data pipelines and streaming applications, enabling reliable data transfer between systems and on‑the‑fly data transformation.

Core concepts include clusters of brokers, topics that act like tables, and partitions that are ordered logs; each partition guarantees order only within itself. Retention policies control how long records are kept.

Producers publish messages to topics, optionally specifying partitions via key hashing or round‑robin, while consumers read messages, track offsets, and belong to consumer groups that ensure each partition is processed by only one consumer in the group.

Kafka brokers handle requests using acceptor and processor threads, store data on disk in segment files with accompanying index files, and manage replication through leader‑follower mechanisms, maintaining an in‑sync replica (ISR) set for durability.

Reliability is configured via acknowledgment levels (acks=0,1,‑1) and replication settings; the leader only acknowledges after ISR followers have persisted data.

Consumer offset management has moved from ZooKeeper to an internal __consumer_offsets topic, enabling fault‑tolerant recovery.

Partition assignment strategies include RangeAssignor, RoundRobinAssignor, and StickyAssignor, each balancing load differently while minimizing reassignment disruption.

Performance optimizations such as sequential disk writes, zero‑copy transfer, batching, and configurable parameters (batch.size, linger.ms, max.in.flight.requests.per.connection) improve throughput and latency.

Code example for custom partitioning logic:

public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    List
partitions = cluster.partitionsForTopic(topic);
    int numPartitions = partitions.size();
    if (keyBytes == null) {
        int nextValue = nextValue(topic);
        List
availablePartitions = cluster.availablePartitionsForTopic(topic);
        if (availablePartitions.size() > 0) {
            int part = Utils.toPositive(nextValue) % availablePartitions.size();
            return availablePartitions.get(part).partition();
        } else {
            return Utils.toPositive(nextValue) % numPartitions;
        }
    } else {
        return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
    }
}

private int nextValue(String topic) {
    AtomicInteger counter = topicCounterMap.get(topic);
    if (null == counter) {
        counter = new AtomicInteger(ThreadLocalRandom.current().nextInt());
        AtomicInteger currentCounter = topicCounterMap.putIfAbsent(topic, counter);
        if (currentCounter != null) {
            counter = currentCounter;
        }
    }
    return counter.getAndIncrement();
}

The article concludes with references and notes on sharing the content.