Understanding Apache Kafka Transactions: Semantics, API Usage, and Performance Considerations

Why Transactions?

Kafka transactions are designed for read‑process‑write streaming applications where both input and output come from asynchronous data streams, such as Kafka topics, and where processing must be exactly‑once, especially in use‑cases like financial debits and credits.

Without transactions, producers may duplicate messages, applications may reprocess input after a crash, and multiple instances ("zombie" instances) can cause duplicate output.

Transactional Semantics

Transactions enable atomic writes to multiple topics and partitions: either all messages in a transaction are committed or none are. Offsets are only considered consumed when they are committed together with the output messages, ensuring an atomic read‑process‑write cycle.

Kafka records the offset commit in an internal __consumer_offsets topic; only when the offset is written does the system consider the input message consumed.

Zombie Fencing

Each transactional producer is assigned a unique transactional.id . The broker tracks an epoch for each ID; if a newer epoch appears, older producers are fenced off and their future writes are rejected, preventing zombie instances.

Reading Transactional Messages

Consumers in read_committed mode receive only non‑transactional messages or messages from committed transactions; they never see messages from open or aborted transactions.

Java Transaction API

The Java client uses initTransactions() to register a transactional ID, beginTransaction() , commitTransaction() or abortTransaction() , and send() calls within the transaction. Consumers are configured with isolation.level=read_committed to enforce exactly‑once processing.

How Transactions Work

When a producer starts a transaction, it registers with the transaction coordinator, which writes state changes to the internal transaction log (a replicated Kafka topic). The coordinator runs a two‑phase commit: first it writes a prepare_commit marker, then a complete_commit marker to the involved partitions.

Only after the commit marker is written are the output records and offset commits visible to consumers.

Practical Transaction Tuning

Transaction overhead is independent of the number of messages; it consists of RPCs to the coordinator, one extra write per partition for the commit marker, and writes to the transaction log. To maximize throughput, batch many records into each transaction.

Longer transaction intervals increase end‑to‑end latency because consumers wait for the commit before delivering messages.

Transactional Consumer Performance

Consumers simply filter aborted transactions and ignore open ones, so in read_committed mode they incur no throughput penalty and require no additional buffering.

Further Reading

For deeper details see the original Kafka KIP, the design document, and the KafkaProducer Javadocs.

Conclusion

The article covered the goals, semantics, and internal mechanics of Kafka transactions, showing how they enable exactly‑once processing for stream‑processing applications and how to tune them for performance.