A Comparative Overview of Transactional Messaging in RocketMQ, Kafka, and Pulsar

Transactional messaging is a mechanism that ensures a set of operations either all succeed or all fail, providing atomicity across distributed systems. This article examines how three widely used open‑source message queue (MQ) platforms—RocketMQ, Apache Kafka, and Apache Pulsar—implement transaction messages, and discusses their underlying concepts, architectures, and use‑case differences.

Background

A transaction is a program execution unit with four ACID properties: Atomicity, Consistency, Isolation, and Durability. In distributed environments, distributed transactions involve multiple participants (services, resource servers, and a transaction manager) located on different nodes. Common solutions include XA (2‑phase commit/3‑phase commit), TCC (Try‑Confirm‑Cancel), and message‑based two‑phase commit.

Message Transaction

Message‑based transactions treat the MQ itself as the coordinator of a two‑phase commit: a local transaction and a message send are bundled together. If the local transaction succeeds, the “half‑message” becomes visible; otherwise it is rolled back. This approach achieves eventual consistency (BASE) rather than strong consistency.

Kafka Transactional Messaging

Kafka’s transaction model is tightly coupled with its idempotent producer feature to provide exactly‑once semantics. A transactional producer is assigned a transactional.id that survives restarts, and each message carries a producerId and sequenceNumber . The transaction coordinator, brokers, and consumers work together to ensure that all messages in a transaction are either committed atomically or aborted, and that consumers only see committed messages (isolation.level=read_committed).

RocketMQ Transactional Messaging

RocketMQ supports distributed transactional messages since version 4.3.0. It uses a two‑phase commit similar to XA: the producer first sends a “half‑message” (invisible to consumers), then executes the local transaction. Depending on the outcome, the half‑message is either committed or rolled back. RocketMQ also provides a transaction status check (reverse lookup) where the broker periodically queries the producer for the transaction state to resolve uncertain cases.

Pulsar Transactional Messaging

Apache Pulsar introduced transaction support in version 2.8.0. Its design resembles Kafka’s exactly‑once model but without a reverse‑lookup mechanism. A transaction ID (TxnID) uniquely identifies a transaction, and a transaction coordinator (TC) running inside the broker manages the transaction lifecycle, timeout handling, and persistence of transaction metadata in a dedicated transaction log. Messages are first written to a transaction buffer and become visible to consumers only after the transaction is committed.

Comparison and Conclusions

All three systems achieve atomicity via a two‑phase commit, storing transactional messages in separate internal topics.

RocketMQ focuses on guaranteeing consistency between a local operation and a message send, using half‑messages and broker‑initiated status checks.

Kafka’s transactions aim at providing exactly‑once delivery for stream processing, relying on idempotent producers, transaction coordinators, and consumer isolation levels.

Pulsar’s transaction model is similar to Kafka’s but differs in implementation details such as client‑side buffering and transaction timeout handling.

Choosing the right MQ for transactional messaging depends on the business requirement: if you need strong consistency between local database updates and message emission, RocketMQ is appropriate; if you need exactly‑once semantics for high‑throughput stream processing, Kafka (or Pulsar) is the better fit.