--- title: "Online vs offline EPLB for MoE serving" description: "Compare offline and online EPLB by traffic drift, measurement windows, expert-weight movement, and the cost of rebalancing a live MoE server." canonical_url: "https://fanout.sh/blog/online-vs-offline-eplb" md_url: "https://fanout.sh/blog/online-vs-offline-eplb.md" last_updated: "2026-08-26" access: "public" --- # Online vs offline EPLB for MoE serving Compare offline and online EPLB by traffic drift, measurement windows, expert-weight movement, and the cost of rebalancing a live MoE server. - Author: Suraj Gaud - Published: 2026-08-26 - Track: Inference engineering - Access: Public - Tags: EPLB, online EPLB, offline EPLB, expert parallelism, mixture of experts, vLLM, DeepSeek, load balancing Online vs offline EPLB is a question about when expert-load measurements become a placement and whether applying that placement can interrupt live serving. Offline EPLB learns from a recorded load matrix and fixes the mapping for the next run. Online EPLB collects the same counts inside the server, then moves expert weights when the traffic distribution changes. Both workflows can use the same packing algorithm, but online rebalancing adds monitoring, coordination, and weight transfer to the serving path. ## Online vs offline EPLB shares one core algorithm Expert parallelism assigns MoE experts to different ranks. If popular experts cluster on one rank, that rank becomes the straggler for every synchronized MoE layer. Fanout's[EPLB guide](/blog/expert-parallelism-load-balancing-eplb)explains how redundant expert copies split that hot load and why each copy consumes GPU memory. The[DeepSeek EPLB implementation](https://github.com/deepseek-ai/EPLB)accepts a matrix of measured expert loads. Each row is a MoE layer and each column is a logical expert. It duplicates heavily loaded experts, then packs physical copies across GPUs. Its hierarchical policy also tries to keep an expert group within a node when topology allows. DeepSeek deliberately leaves load prediction outside the repository. A moving average of historical counts is one suggested input, but the placement function does not know when those counts were collected. That separation creates both workflows. Run the function before deployment for offline EPLB, or wrap it in a live measurement and weight-shuffle loop for online EPLB. ## Offline EPLB freezes a measured placement An offline workflow records routing counts from a representative trace, aggregates them per layer and expert, and computes a logical-to-physical mapping before the next serving run. The model loads each expert onto its assigned ranks at startup. No expert weights move because a new prompt domain becomes popular at noon. This works when traffic is stable enough that yesterday's hot experts are still hot today. It also works for scheduled batch jobs whose prompt mix is known in advance. Offline does not have to mean naive round-robin placement. The load-aware DeepSeek packer can use real production statistics and redundant copies while still applying the result only at startup. Do not confuse that with vLLM's static Zigzag placement. The[Zigzag implementation](https://github.com/vllm-project/vllm/pull/23745)spreads grouped experts without using a live load histogram. Zigzag reported about 8 percent higher queries per minute in one DeepSeek-R1 online-serving benchmark. It is a useful baseline when no representative expert counts exist. Offline load-aware placement can do better for a known workload, but its advantage decays as router preferences drift away from the recorded trace. ## Online EPLB closes the measurement loop [vLLM's expert-parallel deployment guide](https://docs.vllm.ai/en/stable/serving/expert_parallel_deployment/)says the server records load statistics on every forward pass. It aggregates those counts over a window, calculates a new expert mapping at the configured interval, and shuffles weights so the mapping takes effect without restarting the model. The current defaults observe 1,000 engine steps and consider a rebalance every 3,000 steps. Asynchronous weight movement is enabled by default. The original[vLLM EPLB feature](https://github.com/vllm-project/vllm/pull/18343)added dynamic rearrangement for DeepSeek-V2, V3, and R1 models, along with redundant experts. Online EPLB reacts to a coding-heavy morning and a roleplay-heavy evening without an operator producing two placement files. That is its main benefit. It does not make the router balanced. The model still chooses the same logical experts. The runtime changes which physical ranks own copies of those experts. Fanout's[mixture-of-experts routing guide](/blog/mixture-of-experts-routing-explained)covers that logical choice before deployment adds physical placement. ## Convert engine steps into wall-clock lag Window size and step interval are measured in engine steps, not seconds. Their wall-clock meaning changes with batch shape, hardware, and prefill or decode work. Suppose one engine step averages 20 ms. A 1,000-step window covers about 20 seconds, and a 3,000-step interval considers a new placement about once per minute. If long prefills push the average step to 100 ms, the same settings cover 100 seconds and rebalance about every five minutes. The defaults are therefore not a universal response time. Measure step duration on the target pool before claiming that online EPLB reacts within a fixed number of seconds. A short window follows traffic quickly but can duplicate an expert made hot by one burst. A long window produces a steadier map but mixes old traffic into every decision. The interval has a separate job. Recomputing from a 1,000-step window every 100 steps can move weights repeatedly while the evidence is still mostly the same. Choose the window from how many routed tokens make expert popularity stable. Choose the interval from how quickly traffic changes and how much transfer overhead the service can tolerate. ## Work through one traffic shift Suppose an EP8 decode pool sees 8,000 routed tokens per measurement window. Perfect balance would place 1,000 tokens of work on each rank. During coding traffic, rank 3 reaches 1,600 while the mean stays 1,000. Mean divided by maximum gives balancedness of 0.625. An offline coding placement duplicates the hot expert and lowers the maximum to 1,100 on the same trace. Balancedness rises to about 0.909. Now traffic switches domains. A different expert on rank 6 becomes popular, pushing that rank to 1,700 while the old extra copy receives little work. The frozen placement is stale. Online EPLB can observe the new window, move the redundant slot, and attack the new straggler. These numbers are illustrative. The decision comes from production-like per-rank counts plus TPOT and throughput, not balancedness by itself. A brief spike may disappear before a transfer finishes. Persistent domain drift is the case where online adaptation earns its complexity. ## Weight movement is the online tax Online rebalancing must make new expert weights available on their destination ranks. That transfer competes for interconnect bandwidth and can disturb decode latency. The[vLLM wide-EP account](https://vllm.ai/blog/2025-12-17-large-scale-serving)describes a sliding load window, a new logical-to-physical map, and a live weight shuffle. Asynchronous transfer reduces the visible stall but does not make bytes free. It adds buffers, coordination, and a period where old and new mappings must remain consistent. A[vLLM async EPLB evaluation](https://github.com/vllm-project/vllm/pull/43219)measured a 32-GPU GB200 decode setup with 32 redundant experts. At one tested stage and concurrency of 33,600, p99 inter-token latency was 141.1 ms with async transfer and 522.9 ms with synchronous transfer. That result supports async as a safer default, not a universal speedup. The same evaluation reports about 20 seconds of total synchronous stall across 22 rearrangements. Measure latency around rebalance events separately from ordinary steps. A run-wide p50 can hide the exact tail spike EPLB introduces. ## Memory cost does not disappear offline Redundant experts consume the same steady-state HBM whether their mapping was chosen offline or online. vLLM estimates about 2.4 GB per expert-parallel rank for one redundant DeepSeek-V3 expert. That memory can otherwise hold KV cache blocks. Online mode adds movement overhead, but offline mode still has the replica bill. Compare admitted sequences and cache occupancy with the same redundant-expert count. If the workload has little routing skew, neither workflow pays. Even a well-balanced placement cannot recover more than the hot-rank bottleneck that exists. Fanout's[KV cache formula](/blog/kv-cache-formula-llm-inference-memory)converts the lost HBM into context capacity for a specific model and cache dtype. Start by logging balancedness with zero redundant experts. Add replicas only when persistent skew appears beside worse TPOT, throughput, or MoE kernel straggler time. ## Pick offline, online, or a hybrid Use offline EPLB for a fixed batch corpus, stable single-domain traffic, or a deployment where live expert movement is unsupported or too risky. Use online EPLB when request mix changes within the life of one server and the stale mapping causes measurable rank imbalance. A hybrid is often practical. Warm-start from a placement computed on recent production counts, then let the server monitor balancedness without moving weights immediately. Enable live rebalancing after the observation window proves that drift persists. Keep an event log with the old map, new map, bytes moved, transfer duration, and nearby latency percentiles. Test the smallest redundant-expert count first. Online adaptation cannot repay memory that an oversized replica budget removes from useful batching. The choice tracks traffic drift. Freeze the mapping when expert popularity is stable; pay the live transfer tax only when traffic changes faster than the deployment cycle. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md