Introducing RoP 0.2.0: Architecture Upgrade and New Features for RocketMQ on Pulsar

Recently, the Tencent Cloud Middleware team together with the StreamNative community officially released RoP 0.2.0. The new version features a complete architectural redesign that eliminates message loss, duplicate consumption, and partial partition consumption problems.

1. Definition of RoP RoP (RocketMQ on Pulsar) is a pluggable protocol handling plugin that allows Pulsar brokers to understand the native RocketMQ protocol, enabling seamless migration of RocketMQ applications to Apache Pulsar while retaining Pulsar’s powerful capabilities such as compute‑storage separation, multi‑tenant support, cross‑region replication, hierarchical sharding, and lightweight Pulsar Functions.

2. Release of RoP 0.2.0 The 0.2.0 release brings a brand‑new architecture and significant improvements in functionality and stability. New features include ACL authentication, partition expansion for higher concurrent write throughput, and a complete set of native RocketMQ management APIs.

3. Latest Feature Optimizations

(1) RoP ACL Support RoP reuses RocketMQ’s Hook implementation and leverages Pulsar’s built‑in authentication mechanism to provide ACL verification. Users only need to define ACL_ACCESS_KEY and ACL_SECRET_KEY and load them via the existing ACLRPCHook function, minimizing changes to client code.

Example code:

private static final String ACL_ACCESS_KEY = "eyJrZXlJZCI6InJvY2tldG1xLW13bmI3bWFwMjhqZSIsImFsZyI6IkhTMjU2In0." +
            "eyJzdWIiOiJyb2NrZXRtcS1td25iN21hcDI4amVfdGVzdCJ9.BDOjqqY25a6apnZTMZCqg0I0pxVFcqz7fvZbaTqkf5U"; // token
private static final String ACL_SECRET_KEY = "rop";

public static void producer() throws MQClientException {
    DefaultMQProducer producer = new DefaultMQProducer("rocketmq-mwnb7map28je|nit", "ProducerGroupName",
            getAclRPCHook());
    ...
}

static RPCHook getAclRPCHook() {
    return new AclClientRPCHook(new SessionCredentials(ACL_ACCESS_KEY, ACL_SECRET_KEY));
}

Key points of the ACL configuration:

ACL_ACCESS_KEY: Token created at the Namespace level.

ACL_SECRET_KEY: Fixed value used internally by RoP.

(2) MessageID Refactor The original MessageID construction (using PartitionID, LedgerID, EntryID) caused precision loss and could eventually exhaust LedgerID space. RoP 0.2.0 adopts the lightweight broker entry metadata approach (PIP‑70) by adding a 64‑bit index/publish‑time field to the broker protocol header, eliminating the need for client‑side parsing.

Configuration for PIP‑70:

brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor

Note: Broker Entry Metadata is supported from Pulsar 2.8.0 onward.

The refactored MessageID now encodes the following bit fields (total 128 bits):

RESERVED_BITS (1 byte) – prevents negative values.

RETRY_TOPIC_TAG_BITS (1 byte) – marks retry topics.

PULSAR_PARTITION_ID_BITS (10 bits) – supports up to 1024 partitions.

OFFSET_BITS (52 bits) – stores the original offset.

(3) Routing Model Refactor In RoP 0.1.0, routing relied on the RocketMQ “Broker‑Tag + Broker‑Topic‑Seq” model, which cannot be directly mapped to Pulsar’s globally unique partition IDs. RoP 0.2.0 introduces a proxy layer that stores the Topic‑to‑Broker mapping in ZooKeeper, reuses Pulsar’s partition allocation logic, and adds a master‑slave replication mode for high availability.

Example of enabling broker metadata interceptor:

brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor

Replication configuration (suggested even number of nodes):

RoPBrokerReplicationNum=2

With six broker nodes and RoPBrokerReplicationNum=2 , three masters serve traffic while slaves provide failover. The proxy forwards requests to the owner broker when the target topic resides on a non‑owner node.

4. Future Plans The team will continue to enhance RoP with features such as message tracing, message query, back‑tracking, and advanced monitoring. Additionally, they have contributed a plugin that supports arbitrary‑delay messages, extending RocketMQ’s commercial delayed‑message capabilities.

Project repository: https://github.com/streamnative/rop