How Roblox's Cache Sustained 1.38B QPS Beyond Redis Limits

Roblox's caching layer handles 1.38 billion QPS across 6,000+ Redis nodes. When Grow a Garden drove a 10x traffic surge, they survived on existing hardware with a federated architecture and smart efficiency wins.

---

The Redis Ceiling and How They Broke It

Redis clusters hit a practical limit at 400–500 nodes — beyond that, the Gossip protocol's internode chatter consumes too much CPU and network. AWS ElastiCache enforces this same cap.

Roblox's largest backend service needed 100 million QPS from a single logical cluster (up from 10M in three months). Their solution: a federated architecture — a client reverse proxy in front of multiple independent 400-node Redis clusters, presented as one unified service. Today that's 6,000+ nodes across 15+ clusters, scaling horizontally by simply adding more.

---

Zero-Downtime Migration

Migrating live production clusters used a three-stage pipeline — no application changes required:

1. Dual-write to both old and new clusters simultaneously
2. Read switch once TTL-bounded data reaches parity
3. Decommission the original cluster

The proxy layer absorbed all migration complexity transparently.

---

Surviving a 10x Surge on Bare Metal

Grow a Garden shattered gaming CCU records — from 2.8M to 21M concurrent users in three months. Impact: 10x traffic to the largest Redis cluster, all on on-prem hardware with no cloud burst option.

They optimized both their memory-bound Redis fleet (better scheduling, reduced fragmentation) and compute-bound Envoy proxies (tighter autoscaling, leaner health checks). The biggest win: colocating both workloads on the same machines with cgroup isolation — memory-heavy Redis alongside CPU-heavy Envoy — cutting capacity needs by 25%.

---

Reliability at Scale

• Hot key detection: Sampling live traffic to throttle overloaded keys before they cascade
• Partial failure tolerance: Batched requests return partial results instead of all-or-nothing failures
• Chaos engineering: Regular fault injection (node/rack outages, latency) in production
• Client-side guardrails: Rate limiting and retry budgets prevent thundering herds

---

Key Takeaways

1. Federate when a single component can't scale — proxy in front of multiple clusters works for caches, databases, and queues alike
2. Dual-write migrations work beautifully for TTL-bounded data — no complex replication needed
3. Bin-pack complementary workloads — memory-bound + CPU-bound on the same machines with cgroup isolation is free capacity
4. Design batch operations for partial failure from day one

Next up: Roblox is building a multitenant caching service on ValKey for better cost efficiency and resource isolation.

---

Commentary on Roblox's engineering blog post by Sen Li, Anders Persson, Pranish Pantha, and Jeffrey Zhong (March 2026).