Message Deduplication and Exactly‑Once Semantics in RocketMQ

Message middleware is a core component of distributed systems, providing asynchronous processing, decoupling, and traffic shaping. In RocketMQ, the reliability guarantee means that once a message is successfully persisted, it will not be lost, and the system ensures at‑least‑once delivery, i.e., each message is consumed at least once.

Because of this guarantee, the same message may be delivered multiple times in three typical situations: (1) duplicate sending when the producer retries after a network glitch; (2) duplicate delivery when the consumer acknowledges fail due to a network issue; (3) duplicate delivery caused by load‑balancing events such as broker or consumer restarts.

Simple deduplication solution : before processing, check a business table (e.g., SELECT * FROM t_order WHERE order_no='order123' ) and return early if the record already exists. This works in many cases but fails under high concurrency because two threads may read an empty result simultaneously and both proceed.

To handle concurrency, the SELECT ... FOR UPDATE pattern can lock the row, preventing duplicate processing, but it introduces transaction overhead and reduces throughput.

Exactly‑Once semantics aim for a message to be processed exactly once, even if the producer retries. In practice, achieving true exactly‑once requires coordination among the broker, client, and consumer, which is difficult. For database‑centric consumption, a practical approach is to combine the business update with an insert into a dedicated message‑deduplication table within the same transaction. The workflow is: start transaction → insert dedup record (handling primary‑key conflict) → update business table → commit. If the transaction commits, the dedup record guarantees the message was processed; if it fails, the broker will retry.

This transaction‑based method works well when the entire consumption logic relies on a single relational database, but it cannot handle non‑transactional resources (e.g., Redis) or cross‑database operations, and long‑running transactions can cause lock contention.

To remove the transaction dependency, a non‑transactional approach stores deduplication state in a fast store such as Redis. The process inserts a dedup key with a TTL (e.g., 10 minutes) and marks the message as "processing". Subsequent deliveries find the key and are delayed (sent to a retry topic) until the first processing finishes or the TTL expires. This method reduces latency and leverages Redis's built‑in expiration, but sacrifices strong consistency guarantees.

Both approaches solve the majority of duplicate‑delivery problems caused by broker behavior or producer retries. For edge cases involving failures during processing, additional safeguards—such as idempotent downstream RPC calls, graceful consumer shutdown, and monitoring for repeated failures—are recommended.

Finally, the author provides a ready‑to‑use Java implementation (GitHub: https://github.com/Jaskey/RocketMQDedupListener ) that integrates the Redis‑based deduplication logic with RocketMQ's consumer API, requiring only a small configuration change to enable deduplication.