--- title: "EPLB window size and step interval" description: "Turn vLLM's EPLB window size and step interval into sampling duty cycle, wall-clock lag, overlap, and a repeatable tuning procedure." canonical_url: "https://fanout.sh/blog/eplb-window-size-step-interval" md_url: "https://fanout.sh/blog/eplb-window-size-step-interval.md" last_updated: "2026-09-01" access: "public" --- # EPLB window size and step interval Turn vLLM's EPLB window size and step interval into sampling duty cycle, wall-clock lag, overlap, and a repeatable tuning procedure. - Author: Suraj Gaud - Published: 2026-09-01 - Track: Inference engineering - Access: Public - Tags: EPLB, window size, step interval, expert parallelism, mixture of experts, vLLM, load balancing EPLB window size and step interval control different delays. In vLLM, window size chooses how many engine steps feed a placement. Step interval controls when the engine tries a new placement. The defaults are 1,000 and 3,000 steps. That does not mean 1,000 requests or a three-minute interval. Engine-step duration and routed tokens per step change with the live batch. Tune the window for a stable expert-load estimate. Tune the interval for traffic drift and the cost of moving expert weights. ## EPLB window size and step interval control different clocks The current[vLLM expert-parallel guide](https://docs.vllm.ai/en/stable/serving/expert_parallel_deployment/)defines window_size as recorded engine steps and step_interval as the rearrangement frequency. At each recorded step, the runtime collects expert assignment counts from the MoE forward pass. It later sums the window and computes a new physical expert map. Window size is an estimator setting. A short window reacts quickly but gives one burst more influence. A long window mixes more routed tokens and usually produces a steadier popularity estimate. Step interval is an actuation setting. A short interval considers weight movement often. A long interval leaves a stale map in place longer when the request mix changes. The two settings can be equal, separated by a gap, or arranged so adjacent windows overlap. Those three cases do not observe the same traffic. Fanout's[online vs offline EPLB guide](/blog/online-vs-offline-eplb)separates live measurement and weight movement from a placement computed before startup. ## Convert engine steps to wall-clock time Start with the measured duration of an engine step on the target pool. The rough conversion is window seconds equals window steps times mean step seconds. If decode steps average 25 ms, a 1,000-step window spans about 25 seconds. A 3,000-step interval reaches its threshold in about 75 seconds. If mixed prefills raise the mean to 80 ms, the same settings span about 80 seconds and 240 seconds. Configuration did not change, but the response time more than tripled. An average also hides a wide distribution. Large prefills and small decode batches can alternate, so attach timestamps to rearrangement events and measure the actual duration of each sampled window. Do not convert from request rate. Continuous batching can advance one engine step for many active requests, and a single request can remain present across many decode steps. The practical unit is the engine's own step timeline, checked against wall-clock traces from the same workload used to judge placement quality. ## A smaller window creates an unsampled gap The current[vLLM EPLB state implementation](https://github.com/vllm-project/vllm/blob/446c769482dca3278f9d581d72f9b1a05c0690e0/vllm/distributed/eplb/eplb_state.py)avoids recording work that will be overwritten before rearrangement. When window size is smaller than step interval, it records the final window_size steps before the next rearrangement. Earlier steps in that interval are not part of the placement estimate. With the defaults, 2,000 steps pass before the 1,000-step sample begins. The recording duty cycle for placement is 1,000 divided by 3,000, or one third. Suppose traffic shifts just after a rearrangement and stays changed for 1,500 steps. The default placement sample misses that entire period if traffic returns to normal before the final 1,000 steps. Ignoring old steps makes the next map represent recent traffic rather than the whole interval. The gap is useful only when that is the intended sample. It does mean step interval controls both movement frequency and the gap between sampled windows when interval exceeds window. Balancedness logging can cause additional recording near log events. The config notes that logging adds communication overhead, so treat an observation run separately from the steady configuration. ## A larger window overlaps decisions Set window size to 1,000 and step interval to 500. The runtime records continuously because the next rearrangement is always within the window horizon. The first decision can use the latest 1,000 recorded steps once the buffer is full. The next decision arrives 500 steps later, so roughly half of its evidence overlaps the previous window. Overlap smooths placement changes. One burst must persist across more than one interval before it completely replaces the older evidence. The cost is correlation. Two consecutive placements are not independent measurements, and a rapidly changing workload can remain mixed with traffic from the prior regime. Equal window and interval produce back-to-back, mostly non-overlapping windows after startup. That is a useful baseline because each movement decision corresponds to one clear traffic segment. Choose overlap deliberately because it preserves older evidence in the estimator. ## Routed tokens matter more than step count alone The window bounds engine steps, but the load tensor contains expert assignment counts. A step with more routed tokens contributes more counts than a small step. A 1,000-step decode window with 64 live tokens per step represents about 64,000 token positions per MoE layer. A mixed window that includes 100 prefill steps of 4,096 tokens already receives about 409,600 token positions from those prefills before counting the other 900 steps. Those examples use simplified batch sizes, but the imbalance is real: equal step counts do not imply equal statistical mass. Track total routed tokens per window alongside elapsed seconds. A stable estimate needs enough assignments across the prompt domains the pool is expected to serve. Prefill and decode may also favor different experts. If one pool serves both phases, inspect phase composition before blaming window length for a placement that follows the dominant token source. Fanout's[EPLB explainer](/blog/expert-parallelism-load-balancing-eplb)shows how those logical counts become redundant expert copies and a physical GPU map. ## The first rearrangement arrives early The current source initializes rearrangement progress to three quarters of step_interval. The first trigger therefore arrives after about one quarter of the configured interval. At the default interval of 3,000, the first rearrangement is scheduled after about 750 engine steps rather than 3,000. This warm-up behavior matters in short benchmarks. A ten-minute run may contain one early placement and only a few steady-state intervals. Label the first event separately. Its window may contain less representative startup traffic, cache warm-up, model initialization effects, or a smaller sample than later decisions. After a rearrangement, the counter returns to zero and subsequent triggers use the full interval. Do not infer production cadence from the first log line alone. ## Async movement can stretch the interval Current vLLM enables asynchronous EPLB by default. The server can prepare and transfer a new expert layout without imposing the full synchronous stall on the serving path. The step threshold is still not a promise that every rearrangement begins exactly N steps apart. The implementation checks whether the previous asynchronous rearrangement remains in flight. If so, it keeps the counter beyond the threshold and waits instead of starting another transfer. The configured interval is therefore a lower bound on steady trigger spacing when transfers take longer than the available step budget. Record the requested interval, actual trigger step, transfer completion step, and map activation step. Without all four, a delayed response can look like a bad window estimate. Async can hide part of the visible stall. The bandwidth, buffer, and coordination costs of moving weights remain. ## Tune from drift and transfer cost Begin with zero or the smallest practical redundant-expert budget. Log balancedness, per-rank routed tokens, throughput, and tail inter-token latency on a production-like trace. Measure how many routed tokens make the expert ranking stable. Use that as the first window candidate, then translate it into steps for the observed batch mix. Next, find the shortest interval that does not create repeated weight movement or visible tail-latency spikes. The interval should exceed ordinary bursts and remain shorter than persistent domain drift. Run one controlled traffic change. Record when the prompt mix changes, when the sample starts reflecting it, when EPLB proposes a map, and when that map becomes active. Compare window smaller than interval, equal settings, and one overlapping case. Keep the redundant-expert count and traffic trace fixed so the two timing knobs are the only variables. Reject a setting that improves balancedness without improving throughput or latency. Token-count balance is a proxy for useful serving work, not the final objective. The[mixture-of-experts routing guide](/blog/mixture-of-experts-routing-explained)is the next step if the skew begins in the logical router rather than physical placement. Window size decides which recent traffic is evidence. Step interval decides how often that evidence may cause movement. Tune them on separate axes, then verify their interaction on the live step timeline. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md