Design and Implementation of a Redis-Based Delayed Queue

Background In many business processes, actions need to be performed after a delay, such as canceling unpaid orders after 30 minutes, auto‑generating default comments after 48 hours, or canceling undelivered orders after a timeout. Simple polling of the database works for small data volumes but becomes resource‑intensive at scale, prompting the use of delayed queues.

Types of Delayed Queues The article introduces three common implementations: java.util.concurrent.DelayQueue (JDK‑native, easy to use but limited to a single JVM), RocketMQ delayed queue (persistent and distributed but supports only predefined delay levels), and RabbitMQ delayed queue (TTL + DLX, also persistent and distributed but requires messages with the same delay to share a queue).

Considerations for Building a Custom Delayed‑Queue Service When designing a bespoke solution, one must address message storage, real‑time retrieval of expired messages, and high availability.

Redis‑Based Implementation – Version 1.0 The design stores all delayed messages in a Redis hash (Message Pool) keyed by a unique message ID, while 16 sorted sets (ZSETs) act as delayed queues, each entry holding a message ID with its expiration timestamp as the score. A timed task scans the queues, extracts expired messages, and pushes them to a downstream MQ. Message attributes include tags , keys , body , delayTime (or expectDate ), ensuring reliability, persistence, and at‑least‑once delivery.

Redis‑Based Implementation – Version 2.0 To reduce latency, version 2.0 eliminates the 1‑minute polling task and instead uses Java Lock with await / signal to deliver expired messages in real time. The architecture also supports multi‑node deployment: each queue has a dedicated pull‑job thread that fetches expired messages, a worker thread that processes them, and ZooKeeper coordination for dynamic queue re‑allocation.

Main Workflow Upon service start, the node registers with ZooKeeper, obtains its assigned queues, and launches pull‑job threads. Each pull‑job checks its queue for expired messages; if found, the message is handed to a worker for processing and then removed from the queue. Offsets are tracked to avoid loss on failures, and pull‑jobs are recreated when nodes are added or removed.

The article concludes with a call to share the content and contact information for further technical discussions.