Understanding Delay Queues and Their Implementations with JDK, RabbitMQ, Redis, and lmstfy

Delay Queue is a mechanism that stores messages and delivers them to consumers after a specified delay, useful when processing should occur later than the moment of production.

How Delay Queues Work

Messages are initially stored in a queue with a future delivery timestamp; they are moved to a ready queue only when the delay expires, ensuring timed processing.

Typical Use Cases

Cancel unpaid orders after 30 minutes on e‑commerce platforms.

Auto‑confirm a shipment three days after receipt.

Send reminder SMS to users who registered but have not logged in for 30 days.

Notify meeting participants ten minutes before the scheduled start.

Remind customers 30 minutes before a flash‑sale begins.

Common Implementation Methods

Programmatic implementation, e.g., Java's built‑in DelayQueue .

Message‑queue frameworks, e.g., RabbitMQ with the rabbitmq-delayed-message-exchange plugin.

Redis‑based implementation using sorted sets (ZSet).

JDK DelayQueue

Advantages

Easy to use directly in code.

Simple implementation.

Disadvantages

No persistence support.

Not suitable for distributed systems.

RabbitMQ Implementation

RabbitMQ does not natively support delay queues, but the rabbitmq-delayed-message-exchange plugin enables this feature.

Advantages

Supports distributed deployment.

Provides persistence.

Disadvantages

The framework is relatively heavy and requires additional setup and configuration.

Redis Implementation

Redis implements delay queues via a sorted set (ZSet) where the score stores the execution timestamp.

Advantages

Flexible and leverages Redis, a common infrastructure component.

Supports message persistence, improving reliability.

Provides distributed support, unlike JDK's DelayQueue.

High availability by using Redis's own HA mechanisms.

Disadvantages

Requires a continuous polling loop to check for expired messages, consuming a small amount of system resources.

lmstfy

lmstfy is an open‑source Go project from Meitu that builds a lightweight delay queue on top of Redis, consuming minimal resources and proven in high‑traffic production environments.

Features of lmstfy

Basic queue operations: publish, consume, delete.

Message TTL with automatic expiration.

Delay consumption support.

Automatic retry and dead‑letter queue.

Namespace isolation and Prometheus monitoring with Grafana dashboards.

Publish/consume flow control.

Working Principle

When a producer publishes a message with a delay > 0, the message is stored in Redis ZSet (score = absolute delay time). Once the delay expires, a timer moves the message to the ready queue for consumers. If delay = 0, the message goes directly to the ready queue. Consumers always pull from the ready queue. The server periodically checks the ZSet (every second) to transfer expired messages. Messages can be limited by max retry count; exceeding this moves them to a dead‑letter queue, which can be revived or deleted.

Installation and Deployment

Prerequisite

Install Redis with AOF persistence and set the memory eviction policy to noeviction to avoid data loss.

Redis Configuration

# Persistence setting as AOF
appendonly yes
# Memory eviction policy
maxmemory-policy noeviction

Compile Binary

# Download source code
git clone https://github.com/bitleak/lmstfy.git
# Enter project directory
cd lmstfy
# Build the binary (output in ./_build)
make

Start Server

_build/lmstfy-server -c config/demo-conf.toml

Client Usage

Obtain Token

curl --location 'http://127.0.0.1:7778/token/kb-test' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic dGVzdF91c2VyOmNoYW5nZS5tZQ=' \
--data-urlencode 'description=test'

Producer (Go)

package main

import (
    "fmt"
    "github.com/bitleak/lmstfy/client"
)

func main() {
    c := client.NewLmstfyClient("127.0.0.1", 7776, "kb-test", "01HQJ7EQBDZT88JH79BDE4FAYC")
    c.ConfigRetry(3, 50)
    jobId, err := c.Publish("test", []byte("test"), 100, 3, 30)
    if err == nil {
        fmt.Println("Message sent", jobId)
    }
}

Consumer (Go)

package main

import (
    "fmt"
    "github.com/bitleak/lmstfy/client"
)

func main() {
    c := client.NewLmstfyClient("127.0.0.1", 7776, "kb-test", "01HQJ7EQBDZT88JH79BDE4FAYC")
    c.ConfigRetry(3, 50)
    for {
        job, err := c.Consume("test", 6, 3)
        if err != nil {
            panic(err)
        }
        if job != nil {
            fmt.Println(string(job.Data))
            if err1 := c.Ack("test", job.ID); err1 == nil {
                fmt.Println("Message processed and acked")
            }
        }
    }
}

The above demonstrates how to set up and use lmstfy as a Redis‑backed delay queue, offering a lightweight yet robust solution for delayed message processing.