Message Deduplication and Exactly-Once Semantics in RocketMQ

Message middleware is a core component of distributed systems, providing asynchronous processing, decoupling, and traffic‑shaping, but it guarantees only that a message will be delivered at least once ("AT LEAST ONE").

Because the consumer must acknowledge successful processing, a failure before acknowledgment causes the broker to redeliver the same message, leading to duplicate deliveries. RocketMQ can repeat a message with the same messageId in three situations:

Duplicate sending due to network glitches or producer retries after a successful persist.

Duplicate delivery when the consumer’s acknowledgment fails.

Duplicate delivery caused by load‑balancing events such as broker or consumer restarts.

To achieve exactly‑once semantics, the business logic must be idempotent. A simple approach is to check the target table before inserting:

insert into t_order values ...;
update t_inv set count = count-1 where good_id = 'good123';

and then guard the operation with a query:

select * from t_order where order_no = 'order123';
if (order != null) { return; // duplicate }

While this works for low‑concurrency cases, it can still fail when two threads query the same order simultaneously. Using SELECT ... FOR UPDATE inside a transaction locks the row and prevents the race:

select * from t_order where order_no = 'THIS_ORDER_NO' for update;
if (order.status != null) { return; // duplicate }

However, wrapping the whole consumer in a transaction reduces throughput and does not cover non‑relational resources (e.g., Redis) or cross‑database operations.

A more generic solution stores a deduplication record in a separate table (or Redis) and updates its status ("processing", "completed"). The workflow is:

Start transaction.

Insert a deduplication record (handling primary‑key conflicts).

Execute the business logic (e.g., update order).

Commit transaction.

If the consumer crashes before committing, the deduplication record is not persisted, so the message will be redelivered and processed again. The record can also carry an expiration (TTL) to avoid stale "processing" states.

Because the deduplication store does not rely on a relational transaction, Redis can be used for higher performance and built‑in TTL handling:

// Java example using Redis for idempotence
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");
String appName = consumer.getConsumerGroup();
StringRedisTemplate stringRedisTemplate = null; // obtain from Spring context
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener listener = new SampleListener(dedupConfig);
consumer.registerMessageListener(listener);
consumer.start();

This approach solves the majority of duplicate‑delivery problems (broker‑induced, producer‑retries, and concurrent consumption) without modifying business code, though it cannot guarantee idempotence for non‑idempotent external calls (e.g., RPC‑based inventory locks) unless those calls themselves are made idempotent or compensated.

In practice, the solution works for ~99 % of cases; remaining edge cases can be mitigated by ensuring rollback on failure, graceful consumer shutdown, and alerting when non‑idempotent operations repeatedly fail.

Overall, the article provides a practical guide to implementing message deduplication and exactly‑once processing for RocketMQ consumers using both transactional and Redis‑based strategies.