Ensuring No Message Loss in MQ Systems: Interview Guide and Practical Solutions

Interviewers often ask candidates how to guarantee that messages are never lost when using MQ technologies such as Kafka, RabbitMQ, or RocketMQ. The question evaluates both knowledge of middleware fundamentals and the ability to reason about system design.

Case background : In a typical e‑commerce scenario (e.g., JD.com), the transaction service sends a "deduct 100 beans" message to the MQ, and the bean service consumes it to update the user’s balance. This illustrates two common interview angles: system decoupling and traffic‑control.

System decoupling : MQ isolates upstream and downstream services, allowing the transaction service to continue even if the bean service fails, thus improving availability.

Traffic control : MQ smooths traffic spikes (e.g., flash sales) by buffering messages, but it also introduces risks such as message loss, duplication, and backlog.

Message‑loss analysis : The lifecycle is divided into three stages—production, storage, and consumption. In the production stage, a successful ACK from the broker indicates no loss. In the storage stage, brokers replicate messages (usually to at least two nodes) before acknowledging. In the consumption stage, the consumer should acknowledge only after business logic succeeds.

Detection and prevention : Assign a globally unique ID (or monotonically increasing version) to each message, store it in a log table or Redis, and verify its presence before processing. This enables detection of missing messages and supports idempotent consumption.

Idempotency : Implement a message‑log table with fields message_id and status . Before executing business logic, check the log; if the ID already exists, skip processing. This approach works with relational databases or Redis.

Handling backlog : When backlog occurs, scale out consumer instances and increase the number of topic partitions (e.g., in Kafka) so that each consumer has a dedicated partition, ensuring parallel processing.

Global ID generation : Options include database auto‑increment, UUID, Redis atomic counters, or the Snowflake algorithm. Choose based on trade‑offs between simplicity, performance, and availability.

Overall, the interview answer should demonstrate a clear analysis of loss points, a detection mechanism using unique IDs, idempotent processing, and practical scaling tactics for high‑throughput MQ systems.