Implementing Message Queues with Redis: List, Pub/Sub, and Streams

1. Overview of Message Queues

Modern internet applications are increasingly built on distributed system architectures, making message queues a core component for internal communication. A good message queue provides low coupling, reliable delivery, broadcasting, flow control, and eventual consistency.

2. Implementing Message Queues with Redis

2.1 List‑based Queue

Redis List is a simple ordered collection of strings that can be used as an asynchronous queue. Elements can be pushed to the head (LPUSH) or tail (RPUSH) and popped from either end (LPOP, RPOP). By combining push and pop commands you can build various queue patterns.

Common List Commands

LPUSH key value [value ...]
RPUSH key value [value ...]
LPOP key
RPOP key
BLPOP key [key ...] timeout
BRPOP key [key ...] timeout
RPOPLPUSH source destination
BRPOPLPUSH source destination timeout
LLEN key
LRANGE key start stop

Instant Consumption Issue

Using LPUSH/RPOP in a tight while(true) loop to poll for new items wastes CPU cycles. Redis offers blocking commands (BLPOP, BRPOP) that suspend the client until a new element arrives, eliminating unnecessary polling.

Reliable Queue with ACK

Because a List removes a message as soon as it is popped, a consumer crash can cause data loss. The atomic RPOPLPUSH (or its blocking variant BRPOPLPUSH ) copies a message from a working list to a backup list, allowing the application to delete the backup entry only after successful processing, thus providing an acknowledgment mechanism.

127.0.0.1:6379> RPUSH myqueue one
two
three
127.0.0.1:6379> RPOPLPUSH myqueue queuebak
"three"
127.0.0.1:6379> LRANGE myqueue 0 -1
1) "one"
2) "two"
127.0.0.1:6379> LRANGE queuebak 0 -1
1) "three"

2.2 Pub/Sub

Redis also supports a publish/subscribe model where publishers send messages to a channel and all subscribers to that channel receive the message. This model is useful for real‑time broadcasting but does not persist messages.

Pub/Sub Commands

SUBSCRIBE channel [channel ...]

UNSUBSCRIBE [channel ...]

PSUBSCRIBE pattern [pattern ...]

PUNSUBSCRIBE [pattern ...]

PUBLISH channel message

PUBSUB subcommand [argument ...]

Channels are simple keys, while patterns allow wildcard subscriptions (e.g., java.* matches java.framework ).

2.3 Streams

Redis 5.0 introduced the Stream data type, a persistent, append‑only log that provides reliable messaging, consumer groups, and automatic ID generation. Streams overcome the non‑persistence limitation of Pub/Sub and support acknowledgment.

Stream Commands

XADD key ID field value [field value ...]

XTRIM key MAXLEN [~] count

XDEL key ID [ID ...]

XLEN key

XRANGE key start end [COUNT count]

XREAD [COUNT count] [BLOCK ms] STREAMS key [key ...] ID [ID ...]

XGROUP CREATE key groupname ID

XREADGROUP GROUP group consumer [COUNT count] [BLOCK ms] STREAMS key ID [ID ...]

XACK key group ID [ID ...]

XPENDING key group [start end count] [consumer]

XINFO STREAM key

XINFO GROUPS key

Consumer Groups

Consumer groups allow multiple consumers to share the workload of a single stream while each message is delivered to only one consumer in the group. The server tracks a last_delivered_id per group and a Pending Entries List (PEL) for un‑acknowledged messages.

# Create a consumer group that starts from new messages only
127.0.0.1:6379> XGROUP CREATE mystream mygroup $ 
OK
# Read messages as part of the group
127.0.0.1:6379> XREADGROUP GROUP mygroup consumer1 COUNT 1 STREAMS mystream >
1) "mystream"
   2) 1) "1609727806627-0"
      2) 1) "f1"
         2) "v1"
# Acknowledge a processed message
127.0.0.1:6379> XACK mystream mygroup 1609727806627-0
(integer) 1

3. Conclusion

Redis can serve as a lightweight MQ for simple scenarios using List, Pub/Sub, or Streams. Lists are suitable for point‑to‑point queues, Pub/Sub for broadcast, and Streams for durable, consumer‑group‑based processing. However, for heavy‑weight or mission‑critical messaging, dedicated MQ systems such as Kafka or RabbitMQ may still be preferable.