When to Use Redis: Caching, Queues, and Everything Else

Redis is the Swiss Army knife of data stores—you can use it for caching, session storage, message queues, real-time leaderboards, rate limiting, and about fifty other use cases. The problem is that just because you can use Redis for everything doesn’t mean you should.

I’ve seen teams use Redis as their primary database (bad), their only queue (risky), and their caching layer (usually good). The difference between effective Redis usage and a production nightmare often comes down to understanding what Redis is actually good at and where it falls short.

Here’s when to use Redis, when to use something else, and how to avoid the common pitfalls.

What Makes Redis Different

Redis is an in-memory data structure store. The “in-memory” part makes it fast—typical operations take microseconds. The “data structure” part means you get more than just key-value storage: lists, sets, sorted sets, hashes, streams, and more.

This combination makes Redis exceptional at certain problems and terrible at others.

Redis is great for:

• Data that needs to be accessed very quickly (sub-millisecond response times)
• Temporary data that doesn’t need durability guarantees
• Data structures that map naturally to Redis primitives (sorted sets for leaderboards, sets for presence detection)
• Coordinating distributed systems (locks, rate limiting, pub/sub)

Redis is not great for:

• Data that must never be lost (it’s in-memory; hardware failures = data loss)
• Complex queries or joins (it’s not a relational database)
• Datasets larger than available RAM
• Data that doesn’t fit neatly into Redis data structures

Most teams use Redis for caching and session storage. Those are solid use cases, but Redis can do a lot more if you understand the trade-offs.

Caching: The Obvious Use Case

Redis as a cache is a solved problem. You put expensive database queries or API responses in Redis, retrieve them on subsequent requests, and everything gets faster.

import redis
import json

cache = redis.Redis(host='localhost', port=6379, db=0)

def get_user(user_id):
    # Try cache first
    cached = cache.get(f"user:{user_id}")
    if cached:
        return json.loads(cached)

    # Cache miss: fetch from database
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)

    # Store in cache with 1-hour expiration
    cache.setex(
        f"user:{user_id}",
        3600,  # TTL in seconds
        json.dumps(user)
    )
    return user

This pattern works well for:

• Read-heavy workloads where the same data is requested repeatedly
• Expensive computations that don’t change often (aggregations, reports, API responses)
• Session data that needs to be accessed on every request

The mistake teams make is caching everything without thinking about invalidation. If your cache invalidation strategy is “set a TTL and hope,” you’re going to serve stale data eventually.

Better invalidation patterns:

1. Cache-aside with explicit invalidation

def update_user(user_id, data):
    # Update database
    db.execute("UPDATE users SET ... WHERE id = ?", user_id, data)

    # Invalidate cache
    cache.delete(f"user:{user_id}")

2. Write-through caching (update cache and database together)

def update_user(user_id, data):
    # Update database
    db.execute("UPDATE users SET ... WHERE id = ?", user_id, data)

    # Update cache immediately
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)
    cache.setex(f"user:{user_id}", 3600, json.dumps(user))

3. Cache key versioning

# Include a version number in cache keys
CACHE_VERSION = "v2"

def get_user(user_id):
    key = f"{CACHE_VERSION}:user:{user_id}"
    cached = cache.get(key)
    # ...

When you change your data format, increment the version. Old cached data becomes irrelevant automatically.

Session Storage: Better Than Files or Cookies

Storing sessions in Redis is one of the most common use cases, and for good reason: it’s faster than database-backed sessions and more reliable than file-based sessions in multi-server environments.

# Flask example with Redis sessions
from flask import Flask, session
from flask_session import Session
import redis

app = Flask(__name__)
app.config['SESSION_TYPE'] = 'redis'
app.config['SESSION_REDIS'] = redis.Redis(host='localhost', port=6379)
Session(app)

@app.route('/login', methods=['POST'])
def login():
    # Authenticate user
    user = authenticate(request.form['username'], request.form['password'])

    # Store in Redis session
    session['user_id'] = user.id
    session['username'] = user.username
    return redirect('/dashboard')

Redis sessions work well because:

• Fast access on every request (sub-millisecond reads)
• Automatic expiration (sessions expire without manual cleanup)
• Works across multiple servers (no sticky sessions needed)

The catch: if Redis goes down, all sessions are lost and every user gets logged out. For high-availability scenarios, use Redis Cluster or Redis Sentinel with replication.

Rate Limiting and Throttling

Redis is excellent for rate limiting because operations are atomic and very fast. You can implement sliding window rate limiting in a few lines:

def is_rate_limited(user_id, max_requests=100, window_seconds=60):
    key = f"rate_limit:{user_id}"
    pipe = cache.pipeline()
    now = time.time()

    # Remove requests older than the window
    pipe.zremrangebyscore(key, 0, now - window_seconds)

    # Add current request
    pipe.zadd(key, {str(now): now})

    # Count requests in window
    pipe.zcard(key)

    # Set expiration
    pipe.expire(key, window_seconds)

    results = pipe.execute()
    request_count = results[2]

    return request_count > max_requests

This uses a sorted set to track request timestamps. Old requests get removed automatically, and counting is fast even with thousands of requests per user.

This pattern is better than database-backed rate limiting because:

• No database writes on every request
• Atomic operations prevent race conditions
• Automatic cleanup via TTL

Queues: When Redis Works and When It Doesn’t

Redis lists can function as simple queues:

# Producer: add job to queue
cache.lpush("job_queue", json.dumps({
    "task": "send_email",
    "user_id": 12345,
    "params": {...}
}))

# Consumer: process jobs
while True:
    job_data = cache.brpop("job_queue", timeout=5)
    if job_data:
        _, job_json = job_data
        job = json.loads(job_json)
        process_job(job)

This works for:

• Simple background jobs that don’t require guaranteed delivery
• Low-volume queues (thousands of jobs per day, not millions)
• Jobs that can be lost if Redis crashes (non-critical notifications, cache warming)

Redis queues don’t work well for:

• Critical jobs that must never be lost (use RabbitMQ or SQS)
• Jobs requiring complex routing (use RabbitMQ)
• Very high volume (millions of jobs per day; use Kafka or SQS)
• Jobs that need delayed execution (Redis can do this with sorted sets, but dedicated job queues like Sidekiq or Celery are better)

The problem with Redis as a queue is persistence. If Redis crashes before data is flushed to disk, you lose jobs. For email notifications, maybe that’s fine. For payment processing, it’s not.

If you need guaranteed delivery, use Redis Streams instead of lists, or switch to a dedicated message queue.

Redis Streams: A Better Queue

Redis Streams (added in Redis 5.0) are designed for message queues and event streaming:

# Producer
cache.xadd(
    "events",
    {
        "event": "user_signup",
        "user_id": "12345",
        "timestamp": str(time.time())
    }
)

# Consumer group
cache.xgroup_create("events", "email_workers", id='0', mkstream=True)

# Consumer
while True:
    messages = cache.xreadgroup(
        "email_workers",
        "worker_1",
        {"events": '>'},
        count=10,
        block=5000
    )

    for stream, message_list in messages:
        for message_id, data in message_list:
            process_message(data)
            cache.xack("events", "email_workers", message_id)

Streams give you:

• Consumer groups (multiple workers processing the same stream)
• Acknowledgments (jobs aren’t lost if a worker crashes)
• Persistence (with proper Redis persistence configuration)
• Message history (you can replay old messages)

Streams are Redis’s answer to Kafka, but simpler. If you need event sourcing, audit logs, or reliable job processing and don’t want to run Kafka, Redis Streams are a solid choice.

Leaderboards and Counters

Redis sorted sets are perfect for leaderboards:

# Update user score
cache.zadd("game_leaderboard", {"user:12345": 8500})

# Get top 10 players
top_players = cache.zrevrange("game_leaderboard", 0, 9, withscores=True)
# [('user:67890', 9500.0), ('user:12345', 8500.0), ...]

# Get user rank
rank = cache.zrevrank("game_leaderboard", "user:12345")

This is dramatically faster than maintaining leaderboards in SQL:

-- SQL approach: slow on large tables
SELECT user_id, score,
       ROW_NUMBER() OVER (ORDER BY score DESC) as rank
FROM scores
ORDER BY score DESC
LIMIT 10;

For real-time leaderboards, counters, or anything that needs fast ranking/sorting, Redis sorted sets are hard to beat.

Distributed Locks (With Caution)

Redis can implement distributed locks:

def acquire_lock(lock_name, timeout=10):
    lock_key = f"lock:{lock_name}"
    identifier = str(uuid.uuid4())

    # Set if not exists, with expiration
    acquired = cache.set(lock_key, identifier, nx=True, ex=timeout)

    return identifier if acquired else None

def release_lock(lock_name, identifier):
    lock_key = f"lock:{lock_name}"

    # Only delete if we own the lock
    pipe = cache.pipeline(True)
    while True:
        try:
            pipe.watch(lock_key)
            if pipe.get(lock_key) == identifier:
                pipe.multi()
                pipe.delete(lock_key)
                pipe.execute()
                return True
            pipe.unwatch()
            return False
        except redis.WatchError:
            # Retry if modified
            pass

This works for:

• Preventing duplicate jobs (ensure only one worker processes a task)
• Coordinating distributed systems (leader election, resource allocation)
• Avoiding race conditions (atomic counter increments, idempotency checks)

The catch: Redis locks aren’t perfect. In network partition scenarios or Redis failover events, you can lose locks or acquire duplicate locks. For truly critical locking (financial transactions, inventory management), use a proper distributed lock service like Redlock or etcd.

Pub/Sub for Real-Time Notifications

Redis pub/sub is great for broadcasting messages to multiple subscribers:

# Publisher
cache.publish("notifications", json.dumps({
    "type": "new_message",
    "user_id": 12345,
    "message": "Hello!"
}))

# Subscriber
pubsub = cache.pubsub()
pubsub.subscribe("notifications")

for message in pubsub.listen():
    if message['type'] == 'message':
        data = json.loads(message['data'])
        handle_notification(data)

This works for:

• Real-time dashboards (push updates to connected clients)
• Chat applications (broadcast messages to rooms)
• Cache invalidation (notify all servers to clear a cache entry)

The limitation: pub/sub messages aren’t persisted. If a subscriber is offline, it misses the message. For guaranteed delivery, use Redis Streams instead.

When NOT to Use Redis

Redis is not a replacement for:

Your primary database. Redis is in-memory. If it crashes, you lose data. Even with persistence enabled (RDB snapshots or AOF logs), you can lose seconds or minutes of data. Use PostgreSQL, MySQL, or another durable database for data you can’t afford to lose.

Complex queries. Redis doesn’t have joins, aggregations, or a query planner. If you’re doing SELECT ... JOIN ... WHERE ... GROUP BY, stick with SQL.

Data larger than RAM. Redis stores everything in memory. If your dataset is 500GB and you have 32GB of RAM, Redis won’t work. Use a disk-based database.

Long-term storage. Redis is designed for fast access to recent data, not archival storage. For historical logs, analytics data, or anything you need to keep for years, use S3, a data warehouse, or a time-series database.

What We Actually Use

For most projects, we use Redis for:

• Caching (API responses, expensive queries, rendered HTML fragments)
• Session storage (with Redis Sentinel for high availability)
• Rate limiting (API throttling, login attempt tracking)
• Background job queues (non-critical jobs using Redis Streams)
• Real-time features (leaderboards, counters, presence detection)

We don’t use Redis for:

• Primary data storage (use PostgreSQL)
• Critical job queues that can’t lose data (use SQS or RabbitMQ)
• Analytics or reporting (use a data warehouse)
• Long-term log storage (use S3 or Elasticsearch)

Redis is an incredible tool when used for the right problems. The key is understanding that it’s fast and flexible, but not durable or queryable like a traditional database. Treat it as a layer that enhances your architecture, not as a replacement for proven data storage solutions.

Need help designing a caching strategy or implementing Redis for your application? We can help.