Design and Implementation Differences between Pulsar, Kafka, RocketMQ, and InLong‑TubeMQ

This article compares Pulsar, a next‑generation messaging middleware, with Kafka, RocketMQ, and Tencent's open‑source InLong‑TubeMQ from a design and implementation perspective.

1. Cloud‑native Multi‑tenant Design Pulsar natively supports multi‑tenant architecture. Its topic naming follows the pattern persistent://tenant/namespaces/topic , consisting of four parts: domain (persistent or non‑persistent), tenant, namespace, and topic name. This hierarchical naming enables fine‑grained isolation and management, which is more cloud‑friendly than the flat topic structures of Kafka, RocketMQ, and InLong‑TubeMQ.

2. Hierarchical Flow Control Pulsar provides flow control at broker, namespace, and topic levels, covering production, consumption, client connections, and storage quotas. In contrast, Kafka, RocketMQ, and InLong‑TubeMQ have relatively limited flow‑control mechanisms.

3. Compute and Storage Separation Pulsar’s brokers are stateless; they handle request routing and computation, while message storage is delegated to a BookKeeper cluster (bookies). RocketMQ brokers store messages locally using a WAL commit log, and Kafka stores each partition in its own log file. InLong‑TubeMQ stores all partitions of a topic in a single file per broker.

4. Partition and Consumer Model All four systems support partitioned topics, but Pulsar’s partitions are global across the cluster and each partition can be associated with multiple consumers within a subscription group. Kafka, RocketMQ, and InLong‑TubeMQ bind each partition to a single consumer, limiting parallel consumption. Pulsar’s model improves consumption scalability but requires careful flow‑control configuration to avoid broker overload.

5. Message Dispatch Mechanism Pulsar uses a push model: brokers push messages to consumers based on permits (receiveQueueSize). Consumers send flow commands when their permit drops below half. Kafka, RocketMQ, and InLong‑TubeMQ use pull‑based consumption (RocketMQ implements a long‑pull push hybrid). Server‑side message filtering is supported by RocketMQ and InLong‑TubeMQ, while Pulsar’s community version lacks it (TDMQ version provides it).

6. Acknowledgment and Offset Management Kafka, RocketMQ, and InLong‑TubeMQ store a single smallest committed offset per partition, which can cause duplicate processing after failures. Pulsar records the highest fully consumed position (markDeletePosition) and maintains a set of individual deleted message ranges, reducing duplicate consumption but requiring consumers to acknowledge messages in order to avoid memory pressure from many gaps.

7. Metadata Storage Pulsar relies on ZooKeeper for metadata, similar to Kafka, while RocketMQ uses a stateless NameServer, and InLong‑TubeMQ uses a centralized API service. ZooKeeper’s single‑leader write limitation can become a bottleneck in large deployments.

Conclusion Pulsar’s cloud‑native multi‑tenant design, flexible flow control, stateless brokers, and advanced consumption model make it well‑suited for large‑scale, multi‑tenant environments. However, its dependence on ZooKeeper and higher resource consumption for brokers and consumers require careful operational planning. Overall, for new projects within the company, Pulsar is recommended over the other three middleware options.