Message Idempotency and Exactly‑Once Processing in RocketMQ

Message middleware is a core component of distributed systems, offering asynchronous processing, decoupling, and traffic‑shaping, and is typically considered reliable in the sense of "at‑least‑once" delivery: a message that reaches the broker will be consumed at least once.

Because the broker retries delivery until a consumer acknowledges success, the same message can be delivered multiple times, especially when a consumer crashes after processing but before acknowledging. This leads to duplicate consumption, as illustrated with RocketMQ where identical messageId s are repeatedly delivered.

A simple idempotency approach checks the business table before processing, e.g.:

select * from t_order where order_no = 'order123'

if (order != null) { return; // duplicate }

While this works in many cases, it fails under high concurrency because the check and insert are not atomic.

One common fix is to lock the row with SELECT ... FOR UPDATE inside a transaction:

select * from t_order where order_no = 'THIS_ORDER_NO' for update

if (order.status != null) { return; // duplicate }

Transaction‑based idempotency guarantees exactly‑once semantics when the entire consumption logic (including the deduplication check) is wrapped in a single database transaction, but it introduces performance penalties and is limited to relational databases.

To avoid these limitations, a non‑transactional approach stores a dedicated message‑deduplication table with a status column (e.g., "processing", "completed"). Insertion succeeds for the first delivery; subsequent deliveries hit a primary‑key conflict and are delayed, ensuring only one successful processing.

Using Redis as the deduplication store further reduces latency and leverages TTL for automatic expiration of stale "processing" entries, though it sacrifices the strong consistency guarantees of MySQL.

Sample Redis‑based listener code (RocketMQ Java client) demonstrates the minimal change required:

//利用Redis做幂等表

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");

consumer.subscribe("TEST-TOPIC", "*");

String appName = consumer.getConsumerGroup();

StringRedisTemplate stringRedisTemplate = null; // obtain template

DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);

DedupConcurrentListener messageListener = new SampleListener(dedupConfig);

consumer.registerMessageListener(messageListener);

consumer.start();

Both approaches solve the majority of duplicate‑delivery scenarios, but edge cases (e.g., failures after marking a message as "processing") still require careful monitoring, graceful shutdown handling, and possibly manual rollback.

In summary, transaction‑based deduplication offers strong guarantees at the cost of scalability, while a status‑based deduplication table (MySQL or Redis) provides a more flexible, high‑performance solution suitable for many real‑world RocketMQ deployments.