Implementing a Reliable Redis‑Based Delay Queue in Go

Two typical business scenarios are presented: automatically closing unpaid orders after a timeout and sending activation SMS for newly created stores that have not uploaded products within a certain period. Simple periodic table scanning leads to unacceptable latency and high database load.

The article outlines the essential requirements for a delay queue: persistence across restarts, confirmation and retry mechanisms, and precise timing. It argues that professional message‑queue systems like Pulsar or RocketMQ are ideal but introduces a Redis‑based solution to avoid extra middleware.

The core idea is to use Redis sorted sets where each message ID is stored as a member and the delivery timestamp is the score. Additional Redis keys (msgKey, pendingKey, readyKey, unAckKey, retryKey, garbageKey, retryCountKey) are defined to manage message lifecycle, acknowledgments, and retries.

Three Lua scripts are provided to guarantee atomic state transitions:

pending2ReadyScript moves messages whose delivery time has arrived from the pending sorted set to the ready list.

ready2UnackScript pops a message from ready (or retry) and pushes it into the unack sorted set with a retry timestamp, similar to RPOPLPUSH .

unack2RetryScript scans the unack set for messages whose retry time has elapsed, decrements their retry counters, and moves them either back to the retry list or to a garbage set when retries are exhausted.

The Go implementation ( github.com/hdt3213/delayqueue ) offers a DelayQueue type with methods to send delayed messages, start consumption, and handle acknowledgments. A sample program shows how to create a Redis client, register a callback, send ten delayed messages with a one‑hour delay, and start the consumer.

package main

import (
    "github.com/go-redis/redis/v8"
    "github.com/hdt3213/delayqueue"
    "strconv"
    "time"
)

func main() {
    redisCli := redis.NewClient(&redis.Options{Addr: "127.0.0.1:6379"})
    queue := delayqueue.NewQueue("example-queue", redisCli, func(payload string) bool {
        // process message
        return true
    })
    for i := 0; i < 10; i++ {
        err := queue.SendDelayMsg(strconv.Itoa(i), time.Hour, delayqueue.WithRetryCount(3))
        if err != nil { panic(err) }
    }
    done := queue.StartConsume()
    <-done
}

The library ensures that messages are persisted in Redis, so they survive node failures as long as Redis remains healthy and keys are not tampered with. The ack function removes the message from the unack set and deletes its payload key, while nack updates the retry time to trigger immediate re‑delivery.

The main consume loop repeatedly executes pending2Ready , fetches messages via ready2Unack , invokes the user callback, and performs acknowledgment or negative acknowledgment. After processing the ready queue, it runs unack2Retry and a garbage‑collection routine that deletes messages that have exceeded their retry limit.

Overall, the article demonstrates that a Redis‑backed delay queue can provide persistence, atomic state changes, at‑least‑once delivery semantics, and scalable distributed consumption without requiring additional components.