Roblox's caching layer handles 1.38 billion QPS across 6,000+ Redis nodes. When Grow a Garden drove a 10x traffic surge, they survived on existing hardware with a federated architecture and smart efficiency wins.
The Redis Ceiling and How They Broke It
Redis clusters hit a practical limit at 400–500 nodes — beyond that, the Gossip protocol's internode chatter consumes too much CPU and network. AWS ElastiCache enforces this same cap.
Roblox's largest backend service needed 100 million QPS from a single logical cluster (up from 10M in three months). Their solution: a federated architecture — a client reverse proxy in front of multiple independent 400-node Redis clusters, presented as one unified service. Today that's 6,000+ nodes across 15+ clusters, scaling horizontally by simply adding more.
Zero-Downtime Migration
Migrating live production clusters used a three-stage pipeline — no application changes required:
- Dual-write to both old and new clusters simultaneously
- Read switch once TTL-bounded data reaches parity
- Decommission the original cluster
The proxy layer absorbed all migration complexity transparently.
Surviving a 10x Surge on Bare Metal
Grow a Garden shattered gaming CCU records — from 2.8M to 21M concurrent users in three months. Impact: 10x traffic to the largest Redis cluster, all on on-prem hardware with no cloud burst option.
They optimized both their memory-bound Redis fleet (better scheduling, reduced fragmentation) and compute-bound Envoy proxies (tighter autoscaling, leaner health checks). The biggest win: colocating both workloads on the same machines with cgroup isolation — memory-heavy Redis alongside CPU-heavy Envoy — cutting capacity needs by 25%.
Reliability at Scale
- Hot key detection: Sampling live traffic to throttle overloaded keys before they cascade
- Partial failure tolerance: Batched requests return partial results instead of all-or-nothing failures
- Chaos engineering: Regular fault injection (node/rack outages, latency) in production
- Client-side guardrails: Rate limiting and retry budgets prevent thundering herds
Key Takeaways
- Federate when a single component can't scale — proxy in front of multiple clusters works for caches, databases, and queues alike
- Dual-write migrations work beautifully for TTL-bounded data — no complex replication needed
- Bin-pack complementary workloads — memory-bound + CPU-bound on the same machines with cgroup isolation is free capacity
- Design batch operations for partial failure from day one
Next up: Roblox is building a multitenant caching service on ValKey for better cost efficiency and resource isolation.
Commentary on Roblox's engineering blog post by Sen Li, Anders Persson, Pranish Pantha, and Jeffrey Zhong (March 2026).
