Kafka Core Concepts: Basics, Producers & Consumers, Topics, Partitions, and Architecture

Recommended Follow

Top Architect Backend Reply 1024 Special Gift Available

01 Basics

Kafka is a streaming system (also called a message queue) that enables backend services to communicate easily and is a common component in microservice architectures.

02 Producer and Consumer

Producer services (Producer) send messages to Kafka, while consumer services (Consumer) listen to Kafka to receive messages.

A service can act as both producer and consumer.

When a consumer group receives messages, Kafka routes each message to one service within the group, enabling load balancing and easy scaling.

03 Topics

A Topic is the destination address where producers send messages and the listening target for consumers.

A service can listen to and send messages to multiple Topics.

Kafka includes the concept of a consumer group, which is a set of services acting as a single consumer.

If a consumer group receives a message, Kafka routes the message to one service within the group, ensuring load balancing and scalability.

04 Partitions

A Topic is actually composed of multiple queues called Partitions, which allows a Topic to be scaled.

Partitions enable the expansion of Topics.

Producer When sending a message, it is routed to a specific Partition within the Topic.

Consumer Listens to all Partitions and consumes events from all of them.

By default, a producer targets a Topic, and the Topic decides which Partition to place the message in, typically using a round‑robin strategy.

It is also possible to configure a Topic so that messages of the same type are placed in the same Partition, ensuring ordering for that key.

05 Architecture

Kafka runs as a clustered system, with ZooKeeper as a critical component managing Topics and Partitions.

ZooKeeper manages all Topics and Partitions, storing them on physical nodes and maintaining the metadata.

Each Partition has multiple replicas; one replica is elected as the Leader, which receives messages from producers, while the other replicas act as Followers and replicate the Leader’s data.

This replication ensures that each Partition contains the full set of messages, providing fault tolerance—if a node fails, no data is lost.

Overall, this architecture gives Kafka high reliability, scalability, and ordered processing within a Partition.

Welcome to discuss and share your views. If you have questions, feel free to contact me.

Finally, a QR code is provided for readers to obtain a collection of interview questions from major tech companies.

Top Architect Community