Understanding Kafka Consumer Groups, Repartition Triggers, and Consumption Guarantees

Introduction

After previous articles introduced producers, this article explains how Kafka consumers work, the relationship between consumers and consumer groups, when repartition occurs, interaction with Zookeeper, the consumer workflow, and the three consumption guarantees.

Consumer‑Group “Parent‑Child” Relationship

In a consumer group, each partition is consumed by only one consumer, while a consumer may handle multiple partitions; different groups can consume the same partition independently, ensuring the whole topic is fully consumed by the group.

Purpose of Consumer Groups

Consumer groups isolate different business use‑cases on the same topic and bind consumption progress to partitions, enabling smooth rebalancing when consumers join or leave.

When Repartition Is Triggered

Repartition happens when a consumer crashes, a new consumer joins, or the number of partitions in the topic increases, allowing the workload to be redistributed.

Consumer and Zookeeper Interaction

Zookeeper stores consumer offsets, group membership, and partition ownership; consumers read this information to determine which partitions to fetch.

Consumer Workflow

The workflow involves KakfaConsumer , ConsumerConfig , and ConsumerConnector : the config creates a connector, which creates a queue, a fetch thread fills the queue, and a consumer thread processes messages and periodically updates offsets.

Three Consumption Guarantees

Kafka offers at‑least‑once (possible duplicates), at‑most‑once (possible loss), and exactly‑once semantics (no loss or duplication) depending on when offsets are committed relative to processing.

Conclusion

The article covered consumer‑group relationships, repartition triggers, the consumer‑Zookeeper workflow, and the three delivery guarantees, previewing the next topic about server‑side message storage.