Implementation Strategies and Trade‑offs for Delayed Messages in Distributed Message Queues

In distributed asynchronous messaging scenarios, a delayed (or scheduled) message is sent by the producer with the expectation that it will be consumed only after a specified delay or at a particular timestamp, rather than immediately.

Delayed messages are widely used and are typically implemented at the middleware layer, either as a built‑in feature of the MQ or as a separate shared service.

Implementation Schemes

External‑Storage‑Based Schemes

These approaches separate the MQ from a dedicated delay module that stores messages in an external system until they expire, then forwards them to the MQ. Variants differ by the storage used.

Database (e.g., MySQL)

A relational table holds delayed messages. A scanning thread periodically checks for expired rows and delivers them, with the scan interval defining the minimum delay granularity.

CREATE TABLE `delay_msg` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `delivery_time` DATETIME NOT NULL COMMENT '投递时间',
  `payloads` blob COMMENT '消息内容',
  PRIMARY KEY (`id`),
  KEY `time_index` (`delivery_time`)
);

Pros: simple implementation.

Cons: B‑Tree indexes are not optimal for high‑write message workloads.

RocksDB

RocksDB uses an LSM‑Tree, which handles massive writes efficiently. DDMQ’s Chronos module (built on RocketMQ) stores delayed messages in RocksDB, scans them, and re‑injects them into RocketMQ when due.

Pros: LSM‑Tree suits write‑heavy message scenarios.

Cons: heavier solution; developers must implement their own replication and fault‑tolerance for RocksDB data.

Redis

A typical Redis design stores the actual messages in a hash (Message Pool) and maintains 16 ordered ZSETs as delayed queues, where the score is the expiration timestamp. Worker threads poll the ZSETs and deliver due messages.

Pros: ZSET naturally models delayed queues; in‑memory operations are fast.

Cons: Requires careful handling of concurrency (e.g., distributed locks) and load‑balancing across multiple nodes.

Thread‑Based Scanning Issues and Improvements

All the above schemes rely on periodic scanning threads, which waste resources at low load and cause timing inaccuracies at high load. An improvement is to use a wait‑notify mechanism similar to JDK Timer: the scanner waits exactly until the next message’s delivery time, waking early only when a newer earlier message arrives.

Open‑Source MQ Implementations

RocketMQ

Supports 18 predefined delay levels (e.g., 1 s, 5 s, …, 2 h). Delayed messages are stored in a special topic SCHEDULE_TOPIC_XXXX with a dedicated queue per level; a broker later moves them to the target topic.

Pros: low overhead, level‑wise ordering, simple timer management.

Cons: fixed levels limit flexibility; delay messages increase the size of the commit log.

Pulsar

Allows arbitrary delay times. Messages are sent to the target topic, and a separate off‑heap priority queue (per subscription) indexes delayed messages. The broker checks the queue during consumption.

Cons: high memory overhead (one queue per subscription), costly recovery after failures, and storage bloat because delayed messages keep the underlying topic data for the whole delay span.

QMQ

Provides truly arbitrary delays (up to two years) using a two‑level hierarchical time wheel: a coarse wheel on disk (hour‑level logs) and a fine wheel in memory (500 ms slots). Only near‑due messages reside in memory.

Pros: O(1) insert/delete, supports very long delays, memory‑friendly via delayed loading, and stores delayed messages separately from normal traffic.

Cons: Implementation complexity and reliance on disk‑based schedule logs.

The article concludes by summarizing the strengths and weaknesses of each approach, helping readers choose a suitable delayed‑message design for their backend systems.