Master Kafka’s Core Concepts: A Visual Guide to Topics, Partitions, and Architecture
This article visually breaks down Kafka’s fundamental concepts—including producers, consumers, topics, consumer groups, partitions, and cluster architecture—explaining how messages flow, are stored, and achieve load balancing and ordering within a distributed streaming system.
Kafka is a mainstream streaming system with many concepts; the following diagrams clarify its core ideas for a clear mental model.
01 Basics
Kafka is a stream processing system (also called a message queue) that enables backend services to communicate easily and is a common component in microservice architectures.
02 Producers and Consumers
Producer services send messages to Kafka, while Consumer services listen to Kafka to receive messages.
A single service can act as both producer and consumer.
03 Topics
A Topic is the destination address for producers and the listening target for consumers.
A service can listen to and send messages on multiple Topics.
Kafka introduces the concept of a consumer group, which is a set of services acting as a single consumer.
When a consumer group receives a message, Kafka routes the message to one service within the group, aiding load balancing and scaling.
Messages are stored in the Topic’s queue and cannot be modified; they remain until retention policies delete them.
After being stored, the message is delivered to the Topic’s consumers, but it is never removed from the queue.
Messages stay in the queue unchanged; how long they remain depends on Kafka’s configuration.
04 Partitions
A Topic is composed of multiple queues called Partitions, enabling scalability.
Producer routes each message to a specific Partition within the Topic, typically using a round‑robin strategy.
Consumer reads from all Partitions and processes the events.
Messages can be forced into the same Partition to preserve ordering for a particular key (e.g., all messages from user 1).
Only messages within a single Partition are guaranteed to be ordered; across Partitions ordering cannot be ensured.
05 Architecture
Kafka runs as a cluster, with ZooKeeper managing Topics and Partitions.
ZooKeeper tracks the metadata of all Topics and Partitions and maintains the physical Nodes where they reside.
Each Partition has a Leader that receives producer messages and replicates them to Followers, providing fault tolerance.
This replication ensures that all Partition replicas contain the full set of messages, so a node failure does not lead to data loss.
Thus you now have a basic understanding of Kafka’s components; for deeper insights, consult additional resources.
Source: https://timothystepro.medium.com/visualizing-kafka-20bc384803e7
Efficient Ops
This public account is maintained by Xiaotianguo and friends, regularly publishing widely-read original technical articles. We focus on operations transformation and accompany you throughout your operations career, growing together happily.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.