Kafka Interview Guide: Concepts, Architecture, Configuration, and Performance

Kafka is a widely used distributed messaging middleware that enables asynchronous, decoupled communication between services. Understanding its fundamentals is essential for backend developers and interview preparation.

Distributed Messaging Middleware

It provides platform‑independent data exchange, integration of distributed systems, and a message‑queue model that reduces coupling, adds redundancy, improves scalability, smooths traffic spikes, ensures recoverability, ordering, buffering, and asynchronous communication.

Common Interview Topics

Definition and advantages of distributed message middleware

Typical use cases and selection criteria

Key components of Kafka architecture

Producer, Consumer, Consumer Group, Broker, Topic, Partition, Offset, Replication, Record

Kafka Architecture

Key concepts include:

Producer – sends messages to a Topic

Consumer – reads messages from a Topic

Consumer Group – enables parallel consumption while guaranteeing each partition is processed by only one consumer in the group

Broker – server node that stores partitions

Partition – ordered log segment; ordering is guaranteed per partition, not per topic

Offset – unique position of a record within a partition

Replication – multiple copies of a partition for high availability

Command‑Line Tools

Kafka ships with many scripts under the /bin directory, e.g., kafka-console-producer.sh , kafka-console-consumer.sh , kafka-consumer-groups.sh , kafka-topics.sh , and various management and testing utilities.

Producer Configuration

bootstrap.servers – broker addresses

key.serializer / value.serializer

acks – delivery guarantee (0, 1, -1)

retries , retry.backoff.ms

batch.num.messages , linger.ms

compression.type – gzip, snappy, lz4

partitioner.class – custom partitioning for ordering

producer.type – sync or async

Consumer Configuration

bootstrap.servers , group.id

key.deserializer / value.deserializer

enable.auto.commit – manual commit recommended for exactly‑once semantics

auto.offset.reset – latest or earliest

max.poll.records , session.timeout.ms

fetch.max.bytes , request.timeout.ms

Rebalance Mechanism

Rebalance redistributes partitions among consumers when group membership changes, topics are added/removed, or partition counts change. Kafka provides Range and Round‑Robin assignors, and custom assignors can be implemented.

High Availability and Delivery Semantics

Replication with ISR (In‑Sync Replicas) and AR (Assigned Replicas)

Leader election via ZooKeeper; unclean leader election can be enabled for availability at the cost of consistency

Delivery guarantees: at least once , at most once , exactly once (supported from 0.11 with transactions)

Performance Optimizations

Partition‑level concurrency and parallel disk I/O

Sequential append‑only log files per partition

Page cache, pre‑fetching, memory‑mapped files

Binary serialization, compression, batch processing

Lock‑free offset management and Java NIO networking

While the article does not dive into source code, it highlights Kafka’s design choices that are valuable for building scalable, reliable backend systems.