Ensuring Reliable Kafka Messaging: Handling Message Loss, Producer, Broker, and Consumer Configurations

Hello everyone, I am Tom.

Kafka is a widely used messaging framework that connects a high‑performance MQ service between producers and consumers to achieve system decoupling and strong scalability.

You may wonder what happens if any link in the pipeline breaks.

Such a situation is called message loss , which leads to data inconsistency between systems. To address it, we need to consider three dimensions: the producer side, the MQ broker side, and the consumer side.

1. Producer Side

The producer's responsibility is to ensure that messages reach the MQ broker. A callback can be used to determine whether the send operation succeeded, and if it fails, compensation logic is required.

Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback)

Kafka offers several configurable parameters for flexibility:

1.1 acks

This parameter defines how many replica partitions must acknowledge a message before the send is considered successful.

acks=0 – the producer does not wait for any broker response.

acks=1 – the producer waits for the leader partition's acknowledgment.

acks=-1 (or all) – all in‑sync replicas must acknowledge, providing the highest safety at the cost of throughput.

1.2 retries

Specifies the number of retry attempts. If retries are exhausted, the message is temporarily stored on local disk and resent after the broker recovers. A common setting is retries=3 .

1.3 retry.backoff.ms

Defines the pause between retry attempts, typically 300 ms.

Note that a lack of broker response does not always mean a send failure; network jitter may cause a timeout.

Warning: Even if the broker does not respond, the message might still have been accepted.

After these producer safeguards, message delivery is reliable, though duplicate sends can still occur.

2. MQ Broker Side

The broker stores messages and can also lose them, for example when a partition crashes. Kafka mitigates this by using replication.

2.1 replication.factor

Sets the number of replica partitions. When replication.factor > 1 , a follower can be elected as leader if the current leader fails.

2.2 min.insync.replicas

Specifies the minimum number of in‑sync replicas required for a write to be accepted, usually set to a value greater than 1 to ensure durability.

2.3 unclean.leader.election.enable

Controls whether a non‑ISR replica can be elected as leader. Enabling it ( true ) may lead to message loss if the new leader is far behind, so it should be used with caution.

3. Consumer Side

The consumer must fully process messages and then commit its offset. A common pitfall is pulling messages, processing them in a separate thread, and committing offsets before processing finishes; a crash would then cause loss.

The correct flow is: pull → process → commit offset.

Kafka provides the enable.auto.commit flag to control automatic offset commits. To avoid loss, set enable.auto.commit=false and commit manually.

List<String> messages = consumer.poll();
processMsg(messages);
consumer.commitOffset();

If the system crashes after processing but before the offset is committed, the same batch will be re‑pulled, leading to duplicate consumption.

How to Solve Duplicate Consumption and Ensure Data Consistency

Kafka 0.11+ assigns a unique message ID to each record, allowing the broker to filter duplicates and provide idempotent writes.

However, the consumer can still receive the same message multiple times. Two common strategies are:

1) Pull once: commit the offset before processing. This risks data loss if a crash occurs after the commit.

2) Allow duplicate pulls but implement idempotency on the consumer side, ensuring each message is processed only once. This can be achieved by storing a processing flag in a database or Redis and checking it before handling the message.

Recommended reading:

MySQL Open‑Source Tool Collection

What Is a Bloom Filter? Solving High‑Concurrency Cache Penetration

Using Binlog for Cross‑System Data Synchronization