Preventing Message Loss and Achieving Exactly‑Once Semantics in Kafka

When a Kafka producer uses a fire‑and‑forget approach (calling producer.send(msg) without a callback), the method returns immediately but does not guarantee that the message has been successfully persisted, leading to potential loss especially under network jitter or when the message exceeds broker limits.

To avoid such loss, the producer should use the callback‑based API ( producer.send(msg, callback) ) so that the callback can confirm successful commit and trigger retry or compensation logic if needed.

On the consumer side, data loss occurs when a consumer fetches messages, commits its offset, and then crashes before processing the batch; upon restart it resumes from the next offset, skipping the unprocessed messages. The remedy is to disable automatic offset commits and commit offsets only after successful processing of each batch.

Broker‑side loss can happen if an out‑of‑date follower becomes leader (unclean leader election) or if messages remain only in the page cache and the broker crashes before flushing to disk. Preventive measures include disabling unclean leader election, using multiple replicas, and ensuring sufficient replication factor.

Best‑practice settings to minimise loss are: use producer.send(msg, callback) , set acks=all , increase retries , disable unclean leader election, set replication.factor>=3 , set min.insync.replicas>1 , and ensure replication.factor > min.insync.replicas . For consumers, set enable.auto.commit=false and commit offsets manually after processing.

Kafka’s default delivery guarantee is “at‑least‑once”, which prevents loss but may cause duplicates. Exactly‑once semantics are achieved through two mechanisms: idempotent producers and transactional producers.

Idempotence (enabled with enable.idempotence=true ) assigns each producer a PID and a monotonically increasing sequence number per partition; the broker only accepts a message if its new sequence number equals the previous one plus one, discarding duplicates and detecting gaps that indicate possible loss.

Transactions extend idempotence across multiple partitions and sessions. A transactional producer must also enable idempotence, set a transactional.id , and use the transactional API:

producer.initTransactions();
try {
producer.beginTransaction();
producer.send(record1);
producer.send(record2);
producer.commitTransaction();
} catch (KafkaExecption e) {
producer.abortTransaction();
}

When consuming transactional messages, the consumer’s isolation.level determines visibility: read_uncommitted (default) sees all messages, while read_committed only sees messages from successfully committed transactions.

In summary, Kafka provides exactly‑once consumption via idempotent producers (single‑session, single‑partition guarantee) and transactional producers (cross‑partition, cross‑session guarantee), with the latter offering stronger semantics at the cost of lower throughput.