Understanding Kafka Partition Failover When a Broker Goes Offline

The author, a senior architect and author of "RocketMQ Technical Insights," describes a production incident where expanding a Kafka log cluster introduced high latency, and after killing the process of a broker with kill pid , both Java and Go clients reported errors indicating that some partitions had no online leader.

Investigation of the affected partition (dw_test_kafka_0816000, partition 101) using Zookeeper revealed a leader state of -1 and an ISR list containing only a single replica on the now‑offline broker, confirming that the partition’s replication factor was set to 1. Because no other broker held a copy, automatic leader election could not occur.

The root cause was the intentional configuration of a single‑replica topic to reduce inter‑broker traffic under heavy load, which made the system vulnerable to broker outages. The recommended mitigation is to relocate the topic’s partitions away from brokers scheduled for maintenance before shutting them down.

Kafka stores live broker information in Zookeeper under /{namespace}/brokers/brokers/ids as ephemeral nodes. When a broker disappears, its node is removed, triggering the Controller’s onBrokerFailure callback, which subsequently invokes KafkaController.onReplicasBecomeOffline to handle partitions without a leader.

The onReplicasBecomeOffline method groups partitions needing removal, selects those without a leader ( partitionsWithoutLeader ), marks them as OfflinePartition , and drives the state machine to transition them toward OnlinePartition via doHandleStateChanges and sendRequestsToBrokers .

Transitioning from OfflinePartition to OnlinePartition involves assembling the relevant Seq[TopicPartition] , defining the target state, and applying the OfflinePartitionLeaderElectionStrategy . The electLeaderForPartitions method returns sets of successful and failed elections, retrying on recoverable exceptions.

Leader election proceeds by fetching partition metadata from Zookeeper, building a map of TopicPartition → LeaderIsrAndControllerEpoch , comparing controller epochs, and applying the configured election strategy. If unclean.leader.election.enable is false, the first live ISR replica becomes the leader; otherwise, any live replica may be chosen.

Finally, the election results are written back to Zookeeper and propagated to all brokers, completing the failover. The article concludes that moving partitions before broker shutdown prevents write failures and encourages readers to explore the source code for deeper understanding of Kafka’s partition election mechanisms.