Redis vs Memcached for Job Queues
Asynchronous job processing is one of the original reasons Redis was adopted in web stacks. The combination of Lists, Streams, atomic operations, and Lua scripting gives you the primitives every queue library wraps. Memcached was never designed for this and has never been pressed into service for it at scale.
Lists are the original Redis queue
The Redis List is a doubly-linked list with O(1) push and pop at either end. LPUSH adds at the head, RPOP removes from the tail, and the combination is a FIFO queue. The blocking variant BRPOP lets a worker wait without polling, returning the moment a job is available or the timeout expires. For ten years, that was the queue: LPUSH from the producer, BRPOP in a worker loop, JSON-encoded job payload as the value.
The List pattern is at-most-once by default. If the worker crashes between RPOP and the work completing, the job is lost. The reliable variant uses LMOVE (formerly RPOPLPUSH) to atomically move the job from the main queue to a processing-list keyed by worker ID. If the worker crashes, a janitor sweeps the processing-list back into the main queue after a heartbeat timeout. Sidekiq Pro and several other libraries implement variants of this.
The List pattern's limits are visible at scale. Fan-out (one job to many consumers) is awkward (you have to push to multiple lists). Replay (process a job again later) is impossible because RPOP removes it. Backfill from a point in time requires storing job IDs in a separate sorted set. The combined ergonomics improvements is exactly what Redis Streams added in Redis 5.0.
Streams: the 2026 default
Redis Streams are append-only log-structured data. Each entry has a monotonic ID (timestamp plus sequence number), the log is persisted with AOF, and consumer groups let multiple workers process disjoint subsets of messages while tracking what each worker has acknowledged. The model is borrowed directly from Kafka: log, partitions (consumer groups in Redis terms), consumer offsets (pending entries list), and explicit ACK semantics.
The producer call is XADD jobs * type email payload {...}. The consumer call is XREADGROUP GROUP workers worker-1 COUNT 10 BLOCK 5000 STREAMS jobs >. After processing, the worker calls XACK to remove the entry from the group's pending list. If the worker dies before XACK, the entry remains in the pending list and another worker (or the same one) can claim it via XAUTOCLAIM after an idle timeout. This is at-least-once delivery with explicit failure handling, the same guarantee Kafka and SQS offer.
BullMQ for Node.js moved to Streams in version 4 (April 2023) and made it the default backing storage. Sidekiq still defaults to Lists but its Pro tier supports Streams. Python's RQ has experimental Streams support and Celery's Redis backend uses Lists with a pending-set for delivery tracking. The direction of travel is clear: Streams are the queue primitive for new builds, and Lists are kept around for the migration tail.
BullMQ stream consumer pattern
import { Queue, Worker } from "bullmq";
const sendEmail = new Queue("send-email", { connection: { host: "localhost" }});
// Producer
await sendEmail.add("welcome", { userId: 42, to: "ada@example.com" }, {
attempts: 3,
backoff: { type: "exponential", delay: 5000 },
removeOnComplete: 1000,
removeOnFail: 5000,
});
// Consumer (separate process)
new Worker("send-email", async (job) => {
await mailer.send(job.data);
}, {
connection: { host: "localhost" },
concurrency: 10,
});
// Under the hood: XADD on enqueue, XREADGROUP in the worker,
// XACK on completion, XAUTOCLAIM in a background sweeper.Failure modes and what real queue libraries solve
The hard parts of a queue are not enqueue and dequeue. The hard parts are retries, dead letter handling, scheduled jobs, unique-per-job locks, fair scheduling across job classes, observability of in-flight work, and cluster failover behaviour. Every one of these is a feature in a mature queue library, and every one is something you will need to write if you build directly on Redis primitives.
Retries with exponential backoff: you need a delay queue (a sorted set keyed by execute-at-timestamp) and a sweeper that moves due jobs into the main queue. Dead letter queues: a separate stream where permanently failed jobs land for inspection. Scheduled jobs: same sorted-set pattern as retries. Unique-per-job locks: SET NX PX keyed by job dedup ID. Per-class rate limits: token-bucket logic implemented in Lua. Sidekiq alone is over 10,000 lines of Ruby implementing the boring middle of these patterns reliably.
The case for Memcached here is entirely absent. No library targets Memcached as a queue backend, no production deployment of meaningful scale uses Memcached for queueing, and the primitive set (no list, no stream, no atomic multi-key ops, no Lua) makes building it from scratch wildly more complex than just running Redis or Valkey alongside the Memcached you already have for caching. Run them side by side if you want both. Mixing the two does not slow either of them down.
FAQ
What is the difference between Redis Lists and Redis Streams?
Lists are simple FIFO queues consumed with LPUSH / RPOP (or BRPOP for blocking). They are at-most-once unless you implement reliable patterns yourself. Streams are append-only logs with consumer groups, persistent message IDs, explicit acknowledgement, and pending-entry tracking. Streams give you at-least-once delivery, replay, and fan-out. For most production work, use Streams.
Should I use Sidekiq, BullMQ, or roll my own?
Use a library. Sidekiq for Ruby (Lists + atomic Lua), BullMQ for Node.js (Streams under the hood since v4), Celery for Python (Lists, optional Redis Streams backend), RQ for Python. These libraries handle the failure modes (retries, dead letters, scheduled jobs, rate limits per job class) that you would otherwise have to write.
Can Memcached do a queue?
Not really. You could store a comma-separated list as a value, but read-modify-write under contention requires CAS retries, the list size is bounded by Memcached's per-value limit (1MB default), and ordering is brittle. Every production queue stack uses Redis (or RabbitMQ, Kafka, SQS) and not Memcached for a reason.
RPOPLPUSH vs LMOVE?
LMOVE is the modern replacement (Redis 6.2+). RPOPLPUSH still works but is deprecated. LMOVE source destination LEFT RIGHT atomically takes from one list and pushes to another, used in reliable queue patterns where the worker moves a job from the main queue to a processing queue, works on it, then deletes it on success.
Streams vs Kafka, when to switch?
Switch to Kafka when you need durability beyond memory (Streams in Redis can be persisted to AOF but it is not built for petabyte-scale archives), when you need ordered processing across partitions with thousands of consumers, or when you need long retention (days or weeks). Redis Streams are excellent up to roughly tens of millions of messages and hundreds of thousands of ops per second per node.