Understanding Kafka Consumer: Delivery Guarantees, Rebalance Mechanisms, Partition Assignment, and Thread Safety

Author: later_a24d (source: https://www.jianshu.com/p/2932410aa1ec)

Usage Example

public static void main(String[] args) {
    Properties props = new Properties();
    String topic = "test";
    // auto‑offset‑commit
    String group = "test0";
    props.put("bootstrap.servers", "XXX:9092,XXX:9092");
    props.put("group.id", group);
    props.put("auto.offset.reset", "earliest");
    // auto commit
    props.put("enable.auto.commit", "true");
    props.put("auto.commit.interval.ms", "1000");
    props.put("session.timeout.ms", "30000");
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    KafkaConsumer
consumer = new KafkaConsumer<>(props);
    consumer.subscribe(Arrays.asList(topic));
    while (true) {
        ConsumerRecords
records = consumer.poll(100);
        for (ConsumerRecord
record : records) {
            System.out.printf("offset = %d, key = %s, value = %s \n", record.offset(), record.key(), record.value());
            try { Thread.sleep(100); } catch (InterruptedException e) { e.printStackTrace(); }
        }
    }
}

Message Delivery Semantics

At most once: messages may be lost but will never be delivered twice.

At least once: messages will not be lost but may be delivered multiple times.

Exactly once: each message is delivered only once.

Exactly‑Once Implementation

Producer can assign a globally unique ID to each message; the consumer filters duplicates.

Consumer side handling: Process the message first, then commit the offset. If the process crashes before committing, the same message may be re‑consumed (at‑most‑once risk). Commit the offset first, then process. If the process crashes after committing, the message is lost (at‑least‑once risk).

Solution: disable automatic commits, and combine offset commit with message processing inside a transaction (e.g., DB or Redis). On failure, roll back the transaction; on restart, retrieve the stored offset and seek to it.

During rebalance, use listeners to commit offsets manually or seek to stored positions to avoid duplicate consumption.

Consumer Rebalance Triggers

Number of members in the consumer group changes (new consumer joins or leaves).

Number of subscribed topics changes.

Number of partitions for a subscribed topic changes.

Partition Assignment Strategies

RoundRobinAssignor: lists all topic‑partitions and all consumer members, then distributes partitions in a round‑robin fashion.

RangeAssignor: assigns contiguous ranges of partitions to consumers; extra partitions are given to the first few consumers.

// Example of calculating per‑consumer partition counts
numPartitionsPerConsumer = numPartitionsForTopic / consumersForTopic.size();
consumersWithExtraPartition = numPartitionsForTopic % consumersForTopic.size();
// Assignment illustration omitted for brevity

KafkaConsumer Analysis

Thread Safety: KafkaConsumer is not thread‑safe; concurrency control is delegated to the caller.

Request Handling: Requests are sent via a RequestFuture which carries a listener for success/failure callbacks. The compose method adapts futures, and RequestFutureCompletionHandler processes responses.

HeartbeatTask: Periodic heartbeat requests are encapsulated in a DelayedTask . On success the next heartbeat is scheduled; on failure a retry is scheduled.

Poll Method: The core poll method acquires a thread lock, repeatedly calls pollOnce , sends fetches, and returns records. It also triggers delayed tasks (heartbeat, auto‑commit) and ensures single‑threaded access via an AtomicLong guard.

pollOnce Workflow: Ensure the GroupCoordinator is ready. Perform rebalance if needed (JoinGroup/SyncGroup). Update fetch positions for partitions without offsets. Execute delayed tasks (heartbeat, auto‑commit). Return cached records if available; otherwise send fetches and poll the network.

Overall Architecture: SubscriptionState manages topic/partition metadata, Fetch retrieves data, ConsumerCoordinator collaborates with the server’s GroupCoordinator, and ConsumerNetworkClient wraps the low‑level NetworkClient for request transmission.