Kafka Components and Architecture Explained

1. Broker

Each Kafka server is called a Broker; multiple brokers form a Kafka Cluster. A machine can host one or more brokers, all connecting to the same ZooKeeper to constitute the cluster.

2. Topic

Kafka is a publish‑subscribe messaging system. A Topic represents a category of messages; each topic can have multiple consumers. Producers push messages to a topic, and consumers pull them from the topic.

3. Topic and Broker

A broker can host one or more topics, and the same topic can be distributed across multiple brokers within the cluster.

4. Partition Log

Each topic is divided into multiple partitions, each mapping to a logical log file. When a message is published to a partition, the broker appends it to the last segment of the log, which is flushed to disk based on time or message count.

Each partition maintains an ordered, immutable sequence of log records, each assigned a unique offset within the partition.

The broker cluster retains all published messages for a configurable retention period (e.g., 2 days), after which they are discarded.

5. Partition Distribution

Partitions are distributed across multiple brokers for fault tolerance. Each partition has a leader and one or more followers; the leader handles read/write requests, while followers replicate data asynchronously.

Example: For a topic with four partitions, leaders and followers are assigned across brokers to balance load.

6. Producer

Producers generate messages and send them to a specific topic partition, selecting the partition via a configured algorithm or randomly.

7. Consumer

Consumers are organized into consumer groups; each group may consist of processes on different machines.

Messages in a topic can be consumed by multiple consumer groups, but only one consumer within a group processes a given message.

Consumers can subscribe to multiple topics and track offsets per partition, stored in ZooKeeper (pre‑0.9) or the Kafka broker (post‑0.9).

The article also includes diagrams illustrating broker, topic, partition, leader/follower roles, and the overall Kafka architecture.