Message Consumption Patterns and Best Practices in Qunar's QMQ

Yu Zhaohui, who joined Qunar's technology team in 2011, works in the Platform Infrastructure department and has contributed to core components such as a reliable messaging middleware, a task scheduling center, and an order center. He is passionate about system performance optimization, parallelism, and writing clean code.

Qunar has accumulated more than four years of experience building large‑scale transaction systems using a message‑driven architecture. The entire transaction chain is now largely driven by messages, and the company has distilled a set of reusable patterns that are offered as built‑in features of their internal message queue, QMQ.

1. How to handle duplicate messages consumed by a consumer?

1.1 Message consumption modes

There are three typical consumption guarantees: at‑most‑once, at‑least‑once, and exactly‑once.

1.1.1 At‑most‑once – The message is delivered only once regardless of whether the consumer succeeds; suitable for low‑reliability scenarios such as log‑analysis displays where occasional loss is acceptable. This mode offers the highest performance (QMQ’s non‑reliable messages).

1.1.2 At‑least‑once – Most reliable queues adopt this mode. Because the network is unreliable, retries are used, which can introduce duplicate messages (QMQ’s reliable messages).

1.1.3 Exactly‑once – The ideal guarantee, but difficult to achieve solely with the queue. It usually requires additional idempotency checks (QMQ’s idempotent checker).

Since achieving true exactly‑once delivery is hard, systems must accept duplicate messages and adopt appropriate handling strategies.

1.2 Handling approaches

1.2.1 Do nothing – In some cases, duplicate messages have negligible impact, e.g., a cache‑eviction message where repeated deletions only slightly reduce cache‑hit rate.

1.2.2 Business‑level handling – Design business logic to be idempotent. For example, when creating an order, a duplicate order number should be ignored and a success response returned instead of throwing an exception.

1.2.3 Deduplication table – Introduce a deduplication table (in Redis or a database) keyed by unique fields extracted from the message. Upon successful processing, insert a record; on subsequent duplicate messages, check the table and return success without re‑processing. QMQ provides a built‑in idempotent checker that can be configured to use such a table.

Even with deduplication, you must consider transactionality between the business database and the deduplication store; QMQ’s idempotent checker supports transactional deduplication.

2. Message ordering

Ordering is the second major concern for consumers. Implementing strict ordering is costly, and most queues do not provide it out of the box.

2.1 Handling approaches

2.1.1 Do nothing – For scenarios like cache eviction, ordering is irrelevant.

2.1.2 Business‑level handling – Many business processes (e.g., state machines) can tolerate out‑of‑order events by simply rejecting the message and letting it be retried later.

2.1.3 Extra fields

• Non‑strict ordering : Use an updated_time timestamp column; if a message’s timestamp is older than the current record, discard it. This requires reasonably synchronized clocks.

• Strict ordering : Add a version (or oldVersion/newVersion ) field and apply optimistic‑locking updates (e.g., UPDATE tbl SET …, version=version+1 WHERE [email protected] ). If the update affects zero rows, treat it as a conflict and retry.

3. Concurrent updates

Using a version field can also control concurrent updates when a JSON‑style order record is merged with incoming changes. The typical flow is read‑modify‑write with version checking to avoid lost updates.

If a conflict occurs, the system retries; QMQ encapsulates this pattern in a NeedRetryException .

4. Asynchronous processing

When a consumer offloads work to another thread or service, explicit acknowledgments (ack) are required. Misuse examples include creating an extra thread pool unnecessarily or forgetting to call ack after an exception, both of which can lead to message loss or duplicate processing.

5. Batch processing

Processing messages in batches can improve throughput for database inserts, Elasticsearch updates, etc. The typical approach is to buffer incoming messages and consume them when either a size threshold or a time limit is reached.

5.1 Batch when possible – If the consumer can keep up with the producer, batching may be unnecessary; otherwise, use QMQ’s BatchExecutor class.

BatchExecutor automatically adjusts batch size based on processing speed and respects a maximum batch limit.

5.2 Time vs. batch size – QMQ also supports a “first‑to‑trigger” policy where processing occurs when either the batch size or a time interval is reached.

5.3 Multi‑channel batch – For sharded databases, messages can be grouped by shard and batched per channel using QMQ’s multi‑channel batch mode.

The article concludes by summarizing the various consumption patterns and encourages readers to share additional scenarios or questions.