Ensuring Message Processing Once in High-Concurrency Scenarios

This article discusses how to ensure messages are processed only once in high-concurrency scenarios, a common interview question for companies like Alibaba. The author, "the little boy who picks up snails," explains that message duplication can occur due to various reasons such as network issues, broker failures, consumer crashes, or long processing times.

The article proposes a three-layer defense strategy: production-side idempotency (using Kafka's built-in idempotency or transactional messages), broker-side deduplication and persistence (using unique IDs and ensuring message stability), and consumption-side business idempotency (using database constraints, optimistic locking, state machine validation, distributed locks, or deduplication tables).

Additionally, the article suggests monitoring and reconciliation as a fallback solution, including tracking metrics like producer retry rates, consumer duplicate processing alerts, and offset submission delays. The author concludes that while exactly-once semantics exist in some message queues, most scenarios benefit more from at-least-once delivery combined with idempotency design.