Kafka Basics: 15 Key Questions and In‑Depth Answers

Kafka is a distributed streaming platform designed for high‑throughput, low‑latency real‑time data processing. It offers features such as high performance, high throughput, high availability, low latency, scalability, and fault tolerance, making it the preferred choice for message queueing.

Core components include Producer, Consumer, Consumer Group, Broker, Topic, Partition, Replica, Leader, and Follower, each with specific responsibilities in data ingestion, storage, and delivery.

Zookeeper manages cluster metadata, member registration, controller election, and other coordination tasks; however, newer Kafka versions (3.x) aim to remove this dependency via KRaft.

Producer sending modes :

Fire‑and‑forget (asynchronous, highest throughput, lowest reliability)

Sync (blocking, waits for acknowledgment, ensures delivery)

Async with callback (handles errors via callbacks)

Example code for fire‑and‑forget mode:

ProducerRecord<k,v> record = new ProducerRecord<k,v>("this-topic", key, value);
producer.send(record);

Partitioning strategies include round‑robin (default when key is null), key‑based hashing (ensures ordering per key), random, and custom partitioner implementations.

When configuring the number of partitions, balance load and throughput against memory usage, file‑handle limits, latency, and high‑availability impacts; a practical guideline is to keep the total number of leader partitions per broker below 100 × N × F (N = broker count, F = replication factor).

Message ordering is guaranteed only within a single partition; to achieve global ordering, use a single‑partition topic or ensure keys map consistently to the same partition.

Kafka does not support read‑write separation because follower replicas may lag, leading to stale reads.

Replica types: Leader (handles reads/writes) and Followers (replicate data). Deleting messages can be done manually via kafka-delete-records or programmatically, and is also governed by log retention policies (time‑based, size‑based, offset‑based) and log compaction.

Performance optimizations include sequential disk writes, OS page cache (mmap), zero‑copy (sendfile), batch processing, and compression (lz4, snappy, gzip, zstd).

Consumer models are pull‑based, with two consumption patterns: point‑to‑point (each partition consumed by a single consumer in a group) and publish‑subscribe (each consumer group receives all messages). Consumer groups enable horizontal scaling and fault tolerance.

Offsets track the position of each consumer in a partition and are stored in the internal __consumer_offsets topic, which uses a key of <groupId, topic, partition> and a value containing the offset and metadata. Offsets were originally stored in Zookeeper but now reside in Kafka for higher write throughput.

Overall, the article provides a detailed overview of Kafka’s design, configuration, and operational considerations for building robust, scalable streaming applications.