--- title: "Cache hot-key detection window threshold" description: "Size a cache hot-key detector from node spare capacity, reaction delay, queue growth, replica warm-up, and promote-demote hysteresis." canonical_url: "https://fanout.sh/blog/cache-hot-key-detection-window-threshold" md_url: "https://fanout.sh/blog/cache-hot-key-detection-window-threshold.md" last_updated: "2026-10-01" access: "public" --- # Cache hot-key detection window threshold Size a cache hot-key detector from node spare capacity, reaction delay, queue growth, replica warm-up, and promote-demote hysteresis. - Author: Suraj Gaud - Published: 2026-10-01 - Track: System design - Access: Public - Tags: hot keys, cache, detection window, replication, queueing, Redis, load balancing A cache hot-key detection window threshold decides how much evidence the system collects before it reacts. A longer window reduces noise, but it also lets an overloaded owner accumulate more queued work. The threshold cannot be chosen from request share alone. It must fit the node's spare capacity, the detector delay, the time needed to make replicas usable, and the latency budget during that reaction. A useful detector answers a capacity question: how much overload can this shard absorb before the new route is ready? ## Cache hot-key detection window threshold formula Let a node serve at most M requests per second. Its existing cold traffic uses B, so spare capacity S equals M minus B. Suppose a key suddenly adds H requests per second to that node. While the key remains on one owner, backlog grows at max of zero and H minus S requests per second. Let D be total reaction time. D includes evidence collection, the control-loop interval, value copy or cache fill, and routing propagation. The queued work before relief is approximately Q equals max of zero and H minus S, multiplied by D. A rough queue-delay bound is Q divided by M. If the service allows at most L seconds of added queueing, require S to be at least H minus M times L divided by D. This is a planning bound, not a full queueing model. Bursty arrivals, variable request cost, writes, and shared upstream limits need trace replay or simulation. It still exposes the missing unit in many detector settings. A count threshold without a time window cannot be compared with capacity measured in requests per second. ## Work the delay with one hot key Take a cache node that can serve 20,000 requests per second. Cold traffic already uses 14,000, leaving 6,000 requests per second of spare capacity. A new hot key adds 10,000 requests per second. Until routing changes, the node is short by 4,000 requests per second. If detection and promotion take three seconds, the queue gains about 12,000 requests. At 20,000 requests per second, that is roughly 600 milliseconds of service work. To hold this rough bound below 100 milliseconds with the same three-second reaction, S must be at least 9,333 requests per second. The node would need to run below about 53% of its maximum before the surge, even though its normal load looked safe at 70%. Shortening D changes the reserve requirement. At a 500-millisecond reaction time, the same bound needs S of at least 6,000 requests per second, which the node already has. This is why detector cadence is a capacity decision. A slow control loop requires more headroom or an earlier mitigation path. Fanout's[consistent-hash variance guide](/blog/consistent-hash-virtual-node-count-variance)explains why a balanced ring does not balance request cost. The hot-key detector starts where ownership variance stops. ## Measure the resource that saturates Redis defines a hot key as one accessed extremely frequently and notes that one key belongs to one shard. Its[observability guide](https://redis.io/docs/latest/operate/rs/7.8/monitoring/observability/)suggests LFU sampling or operation sampling for diagnosis. Request frequency is useful only when requests have similar cost. One GET of a small value and one range query over a large sorted set do not consume the same CPU or network budget. A detector can track requests, CPU time, response bytes, or a weighted combination. The threshold should use the resource that saturates first. Current Redis source includes hot-key tracking structures for CPU and network metrics, a sampling ratio, and a tracking duration. That layout makes the measurement choices explicit. Sampling lowers observer cost but adds uncertainty. At a one-in-100 sampling ratio, a short burst may finish before the detector collects enough evidence. A Count-Min sketch can estimate frequent keys with fixed memory, but collisions only increase estimated counts. Promotion therefore needs a false-positive budget. Do not set the threshold to "top one key" without a capacity line. The busiest key can be harmless on an idle shard, while the fifth-busiest key can break a shard already close to saturation. ## Window length trades noise for reaction time A tumbling 10-second window may wait almost 10 seconds before the next evaluation. Requiring two consecutive windows makes promotion steadier and doubles the evidence delay. A sliding window updates more often, but it still needs enough samples to separate a real surge from random variation. Smaller buckets improve time resolution and add counter work. An exponentially weighted moving average reacts without storing every bucket. Its decay constant still defines a memory of past traffic and therefore a response time. The[Cornell cache-load study](https://www.cs.cornell.edu/projects/Quicksilver/public_pdfs/hotnets-XIII-final128.pdf)describes detection followed by replication. It reports that Tao's ten-minute replication window could not protect a server from flash crowds. The same study contrasts snapshot reports with streaming heavy-hitter methods. Finer temporal granularity can detect trends faster, but it does not remove replica preparation or routing delay. Choose the window by replaying the shortest surge that must be mitigated. If a damaging spike lasts two seconds, a 60-second detector can report it accurately after the incident is over. Keep a separate slow detector for sustained popularity. One fast path can protect latency while a longer view decides whether the extra replicas should remain. ## Promotion delay belongs in the threshold Detection does not make a replica serve traffic. The system may need to fetch the value, validate its version, install metadata, update routers, and wait for client caches to expire. The[Anna paper](https://dsf.berkeley.edu/jmh/papers/anna_vldb_19.pdf)reports a four-times SLO violation for 15 seconds during hot-key replication and a larger spike during redistribution. In a separate hotspot experiment, Anna promoted a small hot set in about ten seconds. A larger working set took longer because more keys competed for promotion. These measurements show why D should be observed end to end. A fast detector followed by slow data movement still needs capacity for the whole reaction. For read-only values, an application-local cache can make promotion cheap. Redis notes that even a five-second local TTL can remove repeated reads from the shard. Mutable values add invalidation or version checks. A replica that returns stale data is not usable merely because its bytes arrived. The 2026[D-HASH paper](https://doi.org/10.3837/tiis.2026.03.013)adds a guard phase after promotion and keeps writes on the primary while alternating reads. A guard phase increases D by design. Include it in the queue budget instead of treating it as a free correctness detail. ## Replica count sets the steady-state test After promotion, R usable copies can serve a read rate H only if routing and capacity let them share the work. With even routing, the hot-key rate per copy is H divided by R. The primary still carries its cold traffic and any writes pinned to it. For the worked example, four total copies reduce 10,000 hot reads to 2,500 per copy. The primary then sees 16,500 requests per second if cold load stays at 14,000. That leaves 3,500 requests per second on the primary to drain old queued work. A 12,000-request backlog takes about 3.4 seconds to clear at that rate. Promotion can therefore stop backlog growth without instantly restoring latency. Record both time to first relief and time to full recovery. If replica routing is uneven, replace H divided by R with the largest observed replica share. A hashing or locality rule can leave one copy much busier than the mean. Fanout's[weighted rendezvous comparison](/blog/weighted-rendezvous-vs-consistent-hash-ring)covers capacity-aware placement. It does not remove the need to measure a single key's replicated read split. Writes also cap useful replication. Every extra read copy may add invalidation traffic, version metadata, or synchronous write work. ## Add hysteresis before automatic demotion Using the same threshold to promote and demote causes churn near the boundary. Replica warm-up cost is paid repeatedly while routing flips. Use a higher promote threshold and a lower demote threshold. Require the lower rate to persist for a cooldown that exceeds normal burst gaps. The cooldown consumes memory after demand falls, but that memory buys protection against an immediate rebound. Price it against the copy and control-plane work of another promotion. Track false promotions, false demotions, replica lifetime, bytes copied, route updates, and SLO violations during every transition. A detector tuned only on steady Zipf traffic will miss abrupt flash crowds. Include step changes, short spikes, moving hot sets, and write-heavy keys in the replay. Test failures too. A replica that is warming when a node fails may amplify traffic on the remaining owner instead of reducing it. ## A practical detector worksheet Measure M from a latency-constrained load test, not from the maximum throughput before collapse. Record cold load B at the same request mix. For each surge trace, measure incremental hot-key work H. Use CPU or byte-weighted demand when raw request counts hide cost. Break D into detector evidence, evaluation cadence, copy or fill, guard time, routing propagation, and client refresh. Optimize the largest term first. Calculate the backlog bound and compare it with the queue-delay budget. Then replay the trace because the simple arithmetic assumes constant rates. Sweep the promote threshold, window, sampling ratio, replica count, and cooldown. Keep the same traces so lower latency cannot be mistaken for an easier workload. Reject a setting that detects accurately but reacts after the damaging part of the surge. Also reject one that protects latency by promoting so often that copy traffic becomes the new bottleneck. The window and threshold are inseparable from headroom. A hot-key detector is ready when its worst reaction delay, queued work, replica warm-up, and steady-state split all fit one capacity budget. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md