Expert parallelism load balancing with EPLB

Expert parallelism load balancing with EPLB trades GPU memory for a shorter hot-rank bottleneck. It copies popular experts and changes their placement so routed tokens spread across physical ranks.

A redundant DeepSeek-V3 expert costs about 2.4 GB per expert-parallel rank in vLLM, and rebalancing moves weights while the server is running.

The decision starts with measured routing skew. If the hottest rank barely exceeds the mean, the extra copy may remove less time than its memory and movement costs add.

Expert parallelism load balancing starts with the straggler

Expert parallelism places different MoE experts on different GPUs. The router sends each token to its selected experts, then an all-to-all exchange moves activations to the ranks that own those weights.

Fanout's expert parallelism comparison covers why this can reduce weight traffic while adding dispatch and combine communication.

Equal expert counts do not guarantee equal work. Real traffic can route far more tokens to a few logical experts, leaving one GPU busy after its peers finish.

The MoE step waits for that hot rank. More arithmetic on cold ranks cannot hide a persistent straggler.

TensorRT-LLM's EPLB example measures imbalance as hottest-rank tokens minus mean tokens, divided by mean tokens.

If eight ranks receive 8,000 routed tokens, the mean is 1,000. A hottest rank at 1,600 has an imbalance ratio of 0.6, meaning it carries 60 percent more work than the mean.

vLLM logs the inverse view as balancedness: mean tokens divided by maximum tokens. The same distribution scores 1,000 divided by 1,600, or 0.625. Perfect balance scores 1.

Both numbers describe skew, not speedup. Communication, attention, and kernel efficiency remain in the step, so removing 60 percent excess expert work cannot promise a 1.6-times end-to-end gain.

Redundant experts split a hot logical expert

EPLB adds physical copies without changing the model's logical routing decision. Tokens still choose the same learned expert; the runtime can send them to more than one rank holding that expert's weights.

The DeepSeek EPLB implementation estimates expert load, duplicates the hot experts, and packs the resulting replicas onto GPUs to balance total work.

Placement matters as much as replication. Copying two hot experts onto the same rank merely moves the bottleneck.

DeepSeek exposes two policies. Hierarchical balancing keeps expert groups local to nodes when possible, then balances replicas within each node.

Global balancing ignores group boundaries and packs replicas across the full deployment. DeepSeek recommends the hierarchical policy for smaller prefill EP groups and the global policy for larger decode groups.

The split follows communication topology. A placement that evens token counts but sends more traffic across slow links can lose time overall.

Fanout's mixture of experts routing guide works through the logical router before deployment turns those choices into network transfers.

Price the redundant expert in memory

vLLM's EPLB documentation warns that redundant experts compete with KV cache space.

Its local expert footprint depends on MoE layers, bytes per expert, total logical experts, redundant experts, and EP ranks. For DeepSeek-V3, one redundant expert per EP rank is about 2.4 GB.

Across an EP32 deployment, adding 32 redundant experts means each rank holds one extra copy. The cluster pays 32 times 2.4 GB, or 76.8 GPU-GB of aggregate HBM.

The per-GPU bill is more important than the aggregate. Losing 2.4 GB can cut the number of KV-cache blocks, reduce the maximum batch, or remove context headroom.

That creates a direct payback test. Compare throughput and latency with the same offered traffic, then verify whether the balanced run admits fewer sequences because of the smaller cache.

If EPLB reduces the hot-rank time by 8 percent but the lost cache capacity cuts batch throughput by 12 percent, the memory trade failed.

Treat that 8-versus-12 calculation as the method, not a universal threshold. The result changes with cache bytes per token, context lengths, precision, and spare HBM.

Fanout's KV cache memory formula converts the 2.4 GB bill into token capacity for a specific model and cache dtype.

Work through a skew measurement

Suppose an EP8 server records mean rank load of 1,000 tokens per measured step. The hottest rank averages 1,500, so imbalance is 0.5 and balancedness is 0.667.

After adding replicas and rebalancing, the mean remains 1,000 while the hottest rank falls to 1,100. Imbalance becomes 0.1 and balancedness becomes 0.909.

The expert-compute tail above the mean fell from 500 tokens to 100. EPLB removed 80 percent of that excess, but it did not remove 80 percent of total step time.

Now measure the service. Suppose p95 TPOT drops from 30 ms to 25 ms and output throughput rises from 40,000 to 44,000 tokens per second.

That is a 16.7 percent TPOT reduction and a 10 percent throughput gain. If batch capacity and error rate stay unchanged, the replicas are paying rent.

If output throughput instead falls to 38,000 because the smaller KV cache forces lower concurrency, balancedness improved while the serving result worsened.

This is why the go or no-go metric cannot be balancedness alone. It is a diagnostic leading indicator. The decision belongs to SLO-qualified serving capacity and cost.

Choose the statistics window for traffic drift

DeepSeek-V3 detected hot experts from online statistics and adjusted the redundant set periodically, such as every ten minutes, according to its technical report.

vLLM's current defaults observe 1,000 engine steps and rebalance every 3,000. It uses asynchronous weight movement by default to reduce visible latency overhead.

A short window reacts quickly but mistakes noise for a workload change. A long window produces stable placements that can lag after traffic shifts from coding prompts to another domain.

Start by logging balancedness without replicas. Plot it beside request mix, TTFT, TPOT, throughput, and cache occupancy.

Enable EPLB only if low balancedness persists across enough windows to affect serving metrics. Do not add replicas because of one hot batch.

The observation window should contain representative prompt types and enough routed tokens per expert. Sparse counts make the hottest-expert ranking unstable.

Rebalancing cadence also has a bill. Moving weights too often can create latency spikes or consume bandwidth needed by activation exchange.

Measure the event itself. Compare the p99 latency of steps near a rebalance with ordinary steps, even when weight transfer runs asynchronously.

Separate prefill and decode behavior

Prefill routes many tokens from each prompt in one pass. Decode routes a much smaller number per iteration, repeated across the active batch.

Their expert popularity and bottlenecks can differ. Mixing both phases into one heat map may optimize the phase that contributes more tokens rather than the phase breaking the SLO.

The DeepSeek-V3 deployment used EP32 with 32 redundant experts for prefill. Each GPU held eight original experts plus one redundant expert.

Its decode pool used EP320. Each GPU held one expert, while 64 GPUs hosted redundant and shared experts.

Those are architecture-specific production choices, not defaults to copy. They show why one replica count and placement policy need not fit both phases.

If TTFT is the problem, collect prefill statistics and judge prefill latency. If TPOT is the problem, collect decode statistics and judge the streaming path.

Disaggregated serving makes that split explicit. A colocated engine still needs phase-aware measurements because a large prefill batch can dominate the combined window.

When EPLB is worth enabling

First measure rank loads on production-like traffic. Calculate mean-over-max balancedness and the excess-token imbalance ratio for every MoE layer or rank group.

Then find whether the same windows show worse TPOT, lower throughput, or straggler time in MoE kernels. Skew without a serving penalty does not justify more weights.

Budget the per-rank memory before enabling replicas. For DeepSeek-V3 in vLLM, begin with the documented 2.4 GB per redundant expert per EP rank and verify actual allocation on the chosen quantization.

Test the smallest replica count that attacks the persistent hot experts. Compare identical traffic with and without EPLB, including cache blocks, admitted sequences, errors, and rebalance-step latency.

Keep EPLB when the improvement in SLO-qualified capacity exceeds the memory-driven loss in batch capacity and the cost of weight movement.

Turn it back off when balancedness is already near 1, traffic changes too quickly for the window, or 2.4 GB per rank displaces KV cache the workload needs more.

Expert parallelism load balancing earns its memory when it reduces hot-rank stalls after every byte and every rebalance has been charged.