--- title: "MoE GroupGEMM occupancy vs EP size" description: "Derive tokens per expert as EP widens, then separate GroupGEMM shape, weight bandwidth, routing skew, and all-to-all cost." canonical_url: "https://fanout.sh/blog/moe-groupgemm-occupancy-vs-ep-size" md_url: "https://fanout.sh/blog/moe-groupgemm-occupancy-vs-ep-size.md" last_updated: "2026-10-09" access: "public" --- # MoE GroupGEMM occupancy vs EP size Derive tokens per expert as EP widens, then separate GroupGEMM shape, weight bandwidth, routing skew, and all-to-all cost. - Author: Suraj Gaud - Published: 2026-10-09 - Track: Inference engineering - Access: Public - Tags: mixture of experts, GroupGEMM, expert parallelism, GPU occupancy, wide EP, all-to-all, LLM inference MoE GroupGEMM occupancy vs EP size is not a rule that wider expert parallelism creates smaller expert GEMMs. EP decides where experts live. The routed token count decides each GEMM's M dimension. At fixed global concurrency, widening EP leaves the ideal tokens per expert unchanged. At fixed concurrency per GPU, adding EP ranks raises global concurrency, so the ideal M can grow. Wide EP can improve GroupGEMM utilization while storing fewer experts on each GPU. The gain comes from workload shape and aggregate bandwidth, not the EP number by itself. ## MoE GroupGEMM occupancy vs EP size An MoE router assigns each active token to k experts from a model with E routed experts. Under uniform routing, C active tokens create C times k expert assignments. The ideal assignments per expert are therefore M = C times k divided by E. This M is the row count of that expert's input matrix in the grouped feed-forward calculation. NVIDIA's[hardware-friendly LLM design guide](https://developer.nvidia.com/blog/?p=119595)gives the same relationship: GEMM-M equals global concurrency times top-k divided by the expert count. Expert parallel size P does not appear in that formula. P changes the placement: each rank holds E divided by P experts and receives about C times k divided by P assignments. Fanout's[MoE balance batch size guide](/blog/moe-balance-batch-size-vs-expert-parallel-size)derives the related routing counts. GroupGEMM adds the execution question: are those expert matrices large enough to use the GPU well? ## Hold global concurrency fixed Take a model with 256 experts and top-8 routing. Let the decode batch contain 512 active token positions across the whole EP group. The group produces 4,096 expert assignments. Uniform routing gives 16 rows per expert, regardless of whether P is 8 or 32. At EP8, each rank stores 32 experts and receives about 512 assignments. At EP32, each rank stores eight experts and receives about 128 assignments. Each expert still sees M = 16. Wider EP has reduced work per rank, not the matrix height for one expert. This can lower layer latency because more ranks load fewer expert weights in parallel. It can also waste hardware if communication and launch overhead dominate the smaller amount of work on each rank. The[NVIDIA wide-EP report](https://developer.nvidia.com/blog/scaling-large-moe-models-with-wide-expert-parallelism-on-nvl72-rack-scale-systems/)names both sides of the trade. It reports less weight-loading pressure and more aggregate bandwidth while treating high-bandwidth interconnects, custom collectives, and expert load balancing as requirements. ## Hold per-GPU concurrency fixed Now keep 64 active token positions per GPU while increasing P. Global concurrency becomes 64 times P. At EP8, C is 512 and each expert again receives 16 assignments on average. At EP32, C is 2,048 and each expert receives 64. The number of local experts falls from 32 to eight, while M grows from 16 to 64. Each rank still handles about 512 assignments because 64 local tokens choose eight experts. Here, more GPUs expand the global batch without shrinking the per-GPU batch. Each remaining local expert gets a taller matrix. The same NVIDIA design guide says per-GPU concurrency is often capped by KV-cache memory. Sharding expert weights more widely can free memory for more sequences, which raises C again. EP can improve GroupGEMM shape by changing the feasible concurrency, even though EP is absent from M = Ck/E. Do not compare EP8 and EP32 at different global batches and attribute the whole result to placement. Record both global and per-rank active tokens. ## Grouped does not mean one dense GEMM A GroupGEMM kernel batches several expert matrix multiplications into one launch. It avoids launching a separate kernel for every local expert, but the experts retain their own M values. The[Megatron Core MoE guide](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/moe.html)exposes GroupedGEMM as a compute optimization beside separate dispatch and communication controls. Its API passes a tokens-per-expert tensor to the grouped expert layer. That vector, rather than a single batch-size scalar, describes the work presented to the kernel. If eight local experts receive [64, 64, 60, 58, 20, 12, 2, 0] tokens, the launch contains a mix of healthy and tiny matrix multiplications. Grouping reduces launch overhead. It does not turn the last three experts into extra rows for the first expert, because their weights and outputs are different. The[Hugging Face expert-parallel plan](https://github.com/huggingface/transformers/blob/main/docs/source/en/expert_parallelism.md)makes the placement boundary explicit. Its GroupedGemmParallel rule splits weights along the expert dimension so each device loads only its local experts. Routing still decides how many token rows reach each weight shard. ## Routing skew moves the occupancy tail Uniform M is only a planning baseline. Production prompts can concentrate on a subset of experts, leaving some local matrices tall and others nearly empty. Average tokens per expert can look healthy while the grouped kernel spends time on padding, irregular tile counts, or a rank that owns several hot experts. Measure the full tokens-per-expert distribution. Useful summaries include empty-expert fraction, P10, median, P90, maximum M, and the ratio of mean to maximum. Also aggregate by rank. The same expert counts can produce different slowest-rank time when placement puts several hot experts on one GPU. vLLM's[expert-parallel deployment guide](https://github.com/vllm-project/vllm/blob/main/docs/serving/expert_parallel_deployment.md)describes EPLB as a separate mapping layer for this reason. It periodically uses observed load to redistribute expert mappings. EPLB can shorten the slowest rank without changing the router's token choices or the ideal M formula. Fanout's[EPLB explainer](/blog/expert-parallelism-load-balancing-eplb)covers that placement loop. Keep its effect separate from gains caused by a larger global batch. ## Price weight traffic and all-to-all together Small expert M makes feed-forward execution weight-bandwidth heavy. Each expert's matrices must be loaded for few token rows, so the same weight bytes support little arithmetic. Taller M reuses those weights across more rows and raises arithmetic intensity. Fewer local experts can also reduce the set of weights one rank streams during a layer. Wide EP adds activation traffic. Token-expert pairs travel to the owning rank, expert outputs return, and every layer waits for the required collective progress. The useful crossover depends on topology. NVLink within one coherent domain and cross-node InfiniBand do not give the same all-to-all latency or bandwidth. NVIDIA reports up to 1.8 times higher per-GPU output throughput for EP32 than EP8 at 100 tokens per second per user on its tested GB200 NVL72 setup. That number belongs to a system with disaggregated serving, multi-token prediction, custom communication, and load balancing. It is not an EP32 constant. ## Run a controlled EP sweep For each EP size, hold the model, quantization, output-rate target, and request trace fixed. Run one sweep at fixed global concurrency and another at fixed per-GPU concurrency. Record global active tokens, tokens per rank, tokens per expert, local experts per rank, and the two GroupGEMM M distributions. Profile GroupGEMM duration, achieved bandwidth, tensor-core utilization, kernel launches, padding, and empty experts. Pair them with dispatch bytes, collective time, and slowest-rank time. Then raise concurrency until KV memory or the latency objective stops the run. This reveals whether wider EP earns better GEMM shapes by admitting more work. Keep an ungrouped or alternate-kernel baseline when the stack supports one. Tiny M may favor a GEMV-oriented path even when GroupGEMM wins at higher concurrency. MoE GroupGEMM occupancy follows routed rows and weight reuse. EP size changes placement, aggregate bandwidth, communication, and the concurrency the system can hold. Use M = Ck/E as the first check. If a wider configuration wins, identify whether C increased, weight traffic fell, skew improved, or the interconnect absorbed the extra exchange. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md