Explanation of Kafka Components and Architecture
This article provides a comprehensive overview of Kafka's core components—including brokers, topics, partitions, producers, and consumers—explaining their roles, how they interact within a Kafka cluster, and illustrating the architecture with diagrams while also noting recent version changes and related disclaimer information.
Kafka is a distributed publish‑subscribe messaging system. This article introduces its main components: Broker, Topic, Partition, Producer, Consumer, and the overall cluster architecture.
Broker : Each Kafka server is called a broker; multiple brokers form a Kafka cluster, and they can be deployed on one or several machines, all connecting to the same ZooKeeper.
Topic : A topic represents a category of messages; producers publish messages to a topic and consumers subscribe to retrieve them.
Topic and Broker : One broker can host multiple topics, and a single topic can be distributed across many brokers in the same cluster.
Partition log : Each topic is divided into partitions, each mapped to a logical log file composed of ordered, immutable segments. Messages are appended to the latest segment and flushed to disk based on time or size.
Partition distribution : Partitions are replicated across brokers for fault tolerance. Each partition has a leader and one or more followers; the leader handles read/write requests while followers replicate data asynchronously. The set of in‑sync replicas (ISR) is maintained in ZooKeeper, and if a leader fails, a follower is elected as the new leader.
Example: Partition 0 leader is broker 2 with replicas 2, 1, 3; ISR = 2, 1, 3.
Producer : Producers send messages to a specific topic partition, choosing the partition either by a defined algorithm or randomly.
Consumer : Consumers belong to consumer groups; each group can have multiple consumers on different machines, but only one consumer in a group reads a given partition. Offsets are stored in ZooKeeper.
The article also includes architecture diagrams illustrating these components and notes that after version 0.8, consumers no longer communicate directly with ZooKeeper, requiring an updated diagram.
Finally, a disclaimer states that the material is collected from the internet, the copyright belongs to original authors, and the content is for learning purposes only; promotional links and community invitations are also presented.
IT Architects Alliance
Discussion and exchange on system, internet, large‑scale distributed, high‑availability, and high‑performance architectures, as well as big data, machine learning, AI, and architecture adjustments with internet technologies. Includes real‑world large‑scale architecture case studies. Open to architects who have ideas and enjoy sharing.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.