Streams and reliable queue processing

XADD and XREADGROUP, consumer groups, pending entries, XAUTOCLAIM for stalled messages, trimming and choosing between lists and streams.

Adding and reading

XADD events * type order_placed order_id 42 amount 1999
XADD events * type order_paid order_id 42

XLEN events
XRANGE events - +
XREVRANGE events + - COUNT 10
XREAD COUNT 10 BLOCK 5000 STREAMS events $

# only messages newer than a known id, for catch-up after a restart
XREAD COUNT 100 STREAMS events 1758200000000-0

• * lets the server assign the id as milliseconds plus a sequence, which keeps the stream ordered by time.
• $ means only new messages from now on; 0 means everything still in the stream. Confusing the two is how a consumer silently misses a backlog.
• The id is a composite ms-seq string. Compare it as a string only when the millisecond part has the same digit count, or better, let the server do it.
• A stream is a log, not a list: entries stay until you trim them. Without trimming, an active stream grows without bound.

Consumer groups

XGROUP CREATE events workers $ MKSTREAM

# worker 1 pulls up to 10 unread messages
XREADGROUP GROUP workers worker-1 COUNT 10 BLOCK 5000 STREAMS events >

# acknowledge: this is what removes the message from the pending list
XACK events workers 1758200000000-0

# inspect the group
XINFO GROUPS events
XINFO CONSUMERS events workers
XPENDING events workers
XPENDING events workers - + 10 worker-1

# reclaim messages a dead worker left behind after 60s of idle
XAUTOCLAIM events workers worker-2 60000 0-0 COUNT 50

# trim to the last 100k entries when adding
XADD events MAXLEN ~ 100000 * type tick value 1.23

# or trim by time: keep roughly the last day
XTRIM events MINID ~ 1758000000000

The ~ in MAXLEN ~ makes trimming approximate, which is far cheaper because Redis can remove whole radix-tree nodes. Redis performs the exact trim lazily and the length may exceed the limit briefly - that is the intended behaviour.

Reliable processing patterns

• Claim stalled messages in a background loop, not only at startup: a worker can hang for minutes while its in-flight messages age past the timeout.
• Keep the pending list small. A group with a million pending entries is usually a sign that some worker never acknowledges, so check the consumer breakdown before adding capacity.
• Idempotency is still required. A message can be delivered twice: once before the crash and once after it is reclaimed.
• A dead-letter stream is just another stream. On repeated failures, XADD to events:dead with the original id and error, then XACK the original.
• Set a maximum delivery count in your application and route to the dead-letter stream past it, or one poison message will be reclaimed forever.

# find consumers that have stopped acking
XPENDING events workers - + 1000
# then, per consumer, compare delivered and acknowledged counts
XINFO CONSUMERS events workers

Choose a list when each message is a simple job taken by exactly one worker and you do not need replay. Choose a stream when you need consumer groups, acknowledgement, replay from an offset or more than one independent reader.

FAQ

Related

Pub/sub, queues and persistence Everyday patterns: sessions, leaderboards and counters

Last refreshed 2026-09-18.