EPLB window statistics memory overhead

EPLB window statistics memory overhead grows with window steps, MoE layers, physical experts, and bytes per counter. It does not grow with token count directly.

In current vLLM, the persistent load window is an int32 tensor with shape window_size by num_moe_layers by num_physical_experts. Its size is W times L times P times 4 bytes on each rank.

That is separate from the much larger weight cost of redundant experts. Changing window_size and changing num_redundant_experts spend memory in different places.

EPLB window statistics memory overhead formula

The current vLLM EPLB state allocates expert_load_window with three dimensions.

W is the configured window_size in engine steps. L is the number of MoE layers. P is the number of global physical experts, including redundant copies.

The dtype is torch.int32, so each counter occupies four bytes. Persistent window bytes per rank equal W times L times P times 4.

Do not divide this number by expert-parallel size. The source records all physical experts rather than only the experts stored on the local GPU.

The implementation does this so different dispatch methods see consistent global statistics. Every EPLB rank pays for the full count window.

Fanout's EPLB window guide explains which engine steps enter that window. The allocation exists at full size even when only the final part of a longer interval is recorded.

A 1,000-step window can be tens of MiB

Take a model with 48 MoE layers and 128 physical experts. With W = 1,000, the window contains 6,144,000 int32 counters.

Multiply by four bytes to get 24,576,000 bytes, or about 23.4 MiB per rank.

Now take 58 MoE layers, 256 logical experts, and 32 global redundant experts. P is 288, so a 1,000-step window holds 16,704,000 counters.

That is 66,816,000 bytes, or about 63.7 MiB per rank. Across 32 ranks, the replicated persistent windows occupy about 2.0 GiB of aggregate device memory.

Aggregate memory is useful for fleet accounting, but the per-rank 63.7 MiB is the number that competes with the local KV cache and model state.

Double W to 2,000 and both figures double. Changing step_interval alone does not change the allocated window tensor.

The count fits comfortably in many deployments, yet it is large enough to matter near a strict gpu_memory_utilization boundary.

Physical experts include redundant copies

vLLM defines a logical expert as part of the model architecture. A physical expert is one instantiated copy of a logical expert.

If the model has E routed experts and R redundant experts globally, P equals E plus R before any padding or elastic scaling details.

The statistics window therefore grows by W times L times R times four bytes when redundancy increases.

For W = 1,000, L = 58, and R = 32, those extra counter columns consume about 7.1 MiB per rank.

That counter cost is not the main redundancy cost. Each redundant physical expert also needs a copy of its expert weights on some rank.

The vLLM expert-parallel guide gives the expert-storage formula.

It multiplies layers, bytes per expert, and physical experts, then divides by EP ranks.

The incremental redundant-weight cost per rank is roughly L times bytes_per_expert times R divided by EP size. This can reach gigabytes while the added load counters remain in MiB.

Keep the two lines separate in a memory budget. One is an int32 history on every rank. The other is sharded model-weight storage plus movement buffers.

Rearrangement needs temporary statistics too

Persistent allocation is not the peak. During rearrangement, current vLLM creates a logical_expert_load_window before it sums across the step dimension.

Its shape is W by L by (E plus 1), where the extra column catches invalid or padded physical mappings. It uses the same int32 dtype as the persistent window.

With W = 1,000, L = 58, and E = 256, this temporary tensor is about 56.9 MiB per rank.

It coexists with the 63.7 MiB physical-expert window in the example. The statistics path can therefore peak above 120 MiB before smaller maps and framework allocation overhead.

After scatter-add, vLLM sums over W and all-reduces an L by E tensor. For 58 layers and 256 logical experts, that reduced int32 tensor is only about 58 KiB.

Async EPLB clones the reduced tensor so the background worker has a stable snapshot. That clone is also L by E, not another W-step history.

Budget both persistent and temporary tensors. A startup estimate based only on expert_load_window can miss the rearrangement peak.

Per-pass counters and maps are different allocations

The state also keeps expert_load_pass with shape L by P in int32. In the 58 by 288 example, it occupies about 65 KiB.

Each forward pass accumulates routed-token counts there. When the step is recorded, vLLM copies the slice into the circular window and clears the pass tensor.

Physical-to-logical and logical-to-physical maps are integer tensors whose sizes do not scale with W. They belong in a complete EPLB memory profile, but they are not window history.

Current code also creates an expert transfer buffer for rearrangement. Its size follows the expert-weight tensor layout rather than the number of history steps.

This separation matters when a memory profile grows after enabling async movement. Reducing W can shrink the history and its temporary aggregation, but it will not shrink a weight-shaped transfer buffer.

Likewise, reducing redundant experts affects P, expert weights, mappings, and transfers. It changes more than the statistics columns.

Logging changes communication, not the main formula

The log_balancedness option computes average and maximum tokens per rank. vLLM leaves it off by default because the synchronization adds communication overhead.

Enabling logging does not add another W by L by P persistent tensor in the current state. It can make the runtime record near log events and synchronize pass counts more often.

That distinction prevents a common diagnosis error. A latency regression from balancedness logging is not evidence that the load window doubled in memory.

Measure allocated and reserved device memory before enabling EPLB, after state creation, during the first rearrangement, and during steady async movement.

Record latency and collective time beside memory. A 60 MiB tensor may be acceptable while its scatter, sum, or synchronization cadence is not.

The online vs offline EPLB guide separates this live measurement path from a placement computed before startup.

Step interval changes duty cycle instead

When step_interval exceeds window_size, current vLLM records only the final W steps before rearrangement. Earlier steps in the interval do not enter the decision window.

The tensor still has W slices. A 1,000-step window with a 3,000-step interval does not allocate one third of 1,000 slices.

When W exceeds the interval, adjacent decisions can reuse part of the circular history. The allocation remains W slices, while the statistical evidence overlaps.

Choose W for estimator stability and memory. Choose step_interval for movement cadence, traffic drift, and transfer overhead.

Combining them into one "EPLB overhead" knob hides the trade. One setting has a direct linear memory term, while the other mostly changes how often work occurs.

The vLLM forum's top result for this query discusses frequent rearrangement and redundant-weight overhead, but it does not calculate the load-window tensor itself.

That missing calculation is why a copied W = 1,000 can look free until a model with many MoE layers and physical experts is loaded.

Token volume does not set the tensor size

The counters store how many tokens each physical expert processed in a step. A step with 32 routed tokens and a step with 32,000 routed tokens occupy the same L by P slice.

Token volume changes counter values and estimator quality. It does not change the allocated number of counters.

Int32 overflow is a separate correctness question for unusually large accumulated counts. Changing the dtype would also change the four-byte factor in the memory formula.

The current source fixes int32 for expert_load_pass, expert_load_window, and the temporary logical window. Recheck the code when upgrading vLLM rather than preserving four bytes as a timeless constant.

Multiple registered MoE models each receive their own EplbModelState and window. Sum W times L times P times four across those model states for the persistent total.

The MoE routing explainer covers why routed-token counts can differ sharply even though every slice has the same shape.

Compare memory with the performance objective

EPLB balances token counts, but lower imbalance does not guarantee lower decode latency.

The METRO paper reports that token balancing can activate more expert weights during memory-bound decode. Its alternative balances activated experts rather than only tokens.

That result changes the tuning goal. Spending more history memory for a stable token estimate is useful only if the resulting placement improves the target workload.

For each W, record peak device memory, balancedness, prefill latency, decode latency, throughput, and the number of distinct expert replicas activated per batch.

Keep R fixed while sweeping W so the redundant-weight budget does not move under the experiment. Then keep W fixed while sweeping R.

Test one W below the step interval, one equal to it, and one overlapping case. Use the same traffic trace and separate the first warm-up rearrangement from steady events.

Reject a setting that improves the balance metric but reduces useful serving capacity. Statistics exist to choose a placement, not to become the objective.

A practical EPLB memory worksheet

Write down W, L, E, R, EP size, and bytes per expert before starting the server. Compute P = E + R.

Persistent history per rank is W times L times P times four. Temporary logical history is W times L times (E + 1) times four in the current implementation.

Add the L by P pass counters and the integer mapping tables. Then add the expert transfer buffers observed for the selected synchronous or asynchronous communicator.

Price redundant weights separately as L times bytes_per_expert times R divided by EP size. Confirm the actual local expert count when divisibility or padding changes the simple ratio.

Run a profile rearrangement and compare the measured peak with the worksheet. PyTorch reserved memory can exceed live tensor bytes, so leave headroom for allocator behavior and the KV cache.

Finally, repeat after every vLLM upgrade. This article links an exact source revision because tensor shapes, map layouts, and async defaults can change.

Window memory is the easy line to calculate: W times L times P times four bytes per rank. The useful budget keeps that line separate from temporary aggregation, logging collectives, and redundant expert weights.