Design and Implementation of Delayed Message Queues in Distributed Systems

Introduction

Delayed (or scheduled) messages are used in distributed asynchronous messaging when a producer wants a message to be consumed at a specific future time rather than immediately.

The article explores various implementation schemes and their trade‑offs.

Implementation Schemes

1. External Storage Based

Separates the MQ from a dedicated delay module that stores messages in an external storage system until they expire, then delivers them to the MQ.

Database (MySQL) Example

Using a relational table to hold delayed messages and a scanning thread to deliver them when due.

CREATE TABLE `delay_msg` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `delivery_time` DATETIME NOT NULL COMMENT '投递时间',
  `payloads` blob COMMENT '消息内容',
  PRIMARY KEY (`id`),
  KEY `time_index` (`delivery_time`)
);

Advantages: simple implementation. Drawbacks: B+Tree indexes are not optimal for high‑write workloads.

RocksDB

Uses LSM‑tree storage, which is better for massive writes. Example: DDMQ’s Chronos module stores delayed messages in RocksDB and periodically scans for expiration.

Advantages: high write performance. Drawbacks: heavier implementation, requiring custom replication logic.

Redis

Implements a pool of messages (Hash) and multiple ordered queues (ZSET) for different time slices, with workers scanning for due messages.

Advantages: O(log n) insertion with in‑memory speed, ZSET suits delayed queues. Drawbacks: concurrency control and potential duplicate processing across nodes.

2. Timing‑Thread Improvements

Traditional scanning threads waste resources at low load and may miss precise timing at high load. A wait‑notify approach based on the nearest expiration reduces CPU usage and improves accuracy.

Open‑Source MQ Implementations

RocketMQ

Supports 18 fixed delay levels (e.g., 1 s, 5 s, …, 2 h). Messages are stored in a special topic SCHEDULE_TOPIC_XXXX and dispatched to the real topic when the level timer triggers.

Pros: low overhead, ordered delivery per level. Cons: inflexible level configuration, large commit log growth.

Pulsar

Provides arbitrary‑time delayed messages by storing an index in off‑heap memory and using a priority queue. On broker failure, the index is rebuilt.

Cons: high memory usage per subscription, long rebuild times for large time spans, and storage impact due to retaining all messages for the delay period.

QMQ

Offers arbitrary‑time delayed messages using a two‑layer time wheel: a disk‑based hour wheel (schedule log) and an in‑memory 500 ms wheel loaded on demand.

Highlights: O(1) operations, support for multi‑year delays, delayed loading reduces memory pressure, and delayed messages are stored separately from normal traffic.

Conclusion

The article consolidates prevalent delayed‑message designs, compares their strengths and weaknesses, and provides guidance for selecting an appropriate solution based on system requirements.