How to Prevent Duplicate Consumption in RabbitMQ? 5 Practical Solutions for Interviews

Interview Focus Points

Problem Origin : Interviewers want to see if you understand why duplicate consumption happens (network jitter, consumer restart, ACK timeout) rather than just reciting solutions.

Idempotency Design : Recognize that preventing duplicate consumption is essentially achieving business‑level idempotency (unique ID, dedup table, state machine, etc.).

Solution Selection : Choose the most appropriate idempotency technique for the given business scenario instead of defaulting to a single tool like Redis.

Root Cause of Duplicate Consumption

When a consumer finishes processing the business logic but crashes or loses network connectivity before sending an ACK, RabbitMQ does not receive the acknowledgment and therefore re‑delivers the message.

Idempotency Principle

Regardless of how many times a message is delivered, the business effect must be applied exactly once. This is the core idea behind all deduplication solutions.

Solution Comparison

Unique Message ID + Dedup Table – General purpose, medium complexity.

Database Unique Constraint – Simplest, low complexity, works for pure insert‑type operations.

Redis SETNX – High performance, low complexity, suitable for high‑concurrency scenarios.

Optimistic Lock / Version – Medium complexity, fits update‑type operations (stock deduction, balance change).

State Machine – Medium complexity, ideal for order‑flow or ticket‑flow business with clear state transitions.

1. Unique Message ID + Dedup Table (Most Common)

Assign a globally unique messageId to each message. Before processing, query a deduplication table; if the ID exists, skip the message, otherwise process and insert the ID.

@RabbitListener(queues = "order.queue")
public void handleOrder(Message message, Channel channel) throws Exception {
    String messageId = message.getMessageProperties().getMessageId();
    // 1. Check dedup table
    if (duplicateMapper.existsById(messageId)) {
        channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
        return;
    }
    // 2. Business logic
    orderService.createOrder(parseOrder(message));
    // 3. Record dedup
    duplicateMapper.insert(new DuplicateRecord(messageId));
    // 4. ACK
    channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
} catch (Exception e) {
    // Business exception – NACK and requeue
    channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, true);
}

Key Detail : Steps 2 and 3 must run in the same local transaction so that either both succeed or both fail; otherwise the business may execute while the dedup record is not written, leading to another duplicate execution.

Dedup table schema (MySQL example):

CREATE TABLE `msg_duplicate` (
  `message_id` VARCHAR(64) PRIMARY KEY COMMENT 'Message unique ID',
  `create_time` DATETIME DEFAULT NOW() COMMENT 'Processing time'
);

In production, periodically purge old records (e.g., keep only 7 days) to prevent unbounded growth.

2. Database Unique Constraint (Simplest)

If the business operation is inherently an insert (e.g., order creation, user registration), rely on a primary key or unique index to reject duplicate inserts.

@RabbitListener(queues = "order.queue")
public void handleOrder(Message message, Channel channel) {
    Order order = parseOrder(message);
    try {
        orderMapper.insert(order); // unique index on order_no throws DuplicateKeyException on repeat
    } catch (DuplicateKeyException e) {
        log.warn("Duplicate message, skip: {}", order.getOrderNo());
    }
    channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}

This approach is "simple, brutal, and reliable" as long as the table already has a natural unique key.

3. Redis SETNX (High‑Performance)

For high‑throughput scenarios, use Redis SETNX (set if not exists) to perform a fast dedup check with an expiration to avoid key buildup.

@RabbitListener(queues = "order.queue")
public void handleOrder(Message message, Channel channel) {
    String messageId = message.getMessageProperties().getMessageId();
    Boolean isFirst = stringRedisTemplate.opsForValue()
        .setIfAbsent("msg:dedup:" + messageId, "1", 24, TimeUnit.HOURS);
    if (Boolean.FALSE.equals(isFirst)) {
        channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
        return; // already processed
    }
    // Business logic
    orderService.createOrder(parseOrder(message));
    channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}

Pitfall : There is a time gap between the successful SETNX and the execution of business logic. If the business fails, the Redis key already marks the message as processed, causing loss. The fix is to wrap the check and business execution in a Lua script for atomicity, or fall back to the first solution with a local transaction.

4. Optimistic Lock / Version (Update‑Type Operations)

When the operation updates existing rows (e.g., stock deduction, balance change), include a version column in the WHERE clause. If the update affects zero rows, it indicates either a duplicate consumption or insufficient stock, and no retry is needed.

-- Deduct stock with version check
UPDATE goods_stock
SET stock = stock - 1,
    version = version + 1
WHERE goods_id = 1001
  AND version = 5
  AND stock > 0;

If the affected row count is 0, treat it as a duplicate or out‑of‑stock situation.

5. State Machine (Order Flow)

Model the business as a finite state machine where each state can only transition forward (e.g., Pending → Paid → Shipped ). Persist the current state; processing a message checks the state and moves it forward only if the transition is valid, guaranteeing idempotent progression.

High‑Frequency Follow‑up Questions

Can RabbitMQ itself prevent duplicate delivery? No. Its ACK mechanism guarantees "at least once" delivery, not "exactly once". Business‑level idempotency is mandatory.

What if the dedup table and business table reside in different databases? Distributed transactions (e.g., Seata) are possible but add complexity. A practical approach is to write the dedup record first (with a status field) and later compensate the business table, or combine Redis SETNX with a DB unique constraint.

Will the dedup table become a bottleneck under massive traffic? Yes. A layered approach works: Redis for fast first‑level dedup, falling back to the DB table as a second‑level safeguard.

Common Interview Variants

"How to solve RabbitMQ duplicate consumption?"

"How to guarantee message idempotency?"

"How to implement the exactly‑once semantics of a message queue?"

Memory Mnemonic

防重复 = 业务幂等 – RabbitMQ only guarantees "at least once"; achieving exactly‑once requires business‑level idempotency.

Summary

RabbitMQ can re‑deliver messages when the consumer crashes before ACK, so the only reliable solution is to make the business logic idempotent; the most widely used pattern is a globally unique message ID plus a deduplication table, while database unique constraints, Redis SETNX, optimistic locking, and state‑machine designs serve as scenario‑specific alternatives.