Expert parallelism vs tensor parallelism for MoE

Expert parallelism vs tensor parallelism is a choice about how to cut the expert weights of a mixture-of-experts model, and the cut decides which communication pattern runs inside every MoE layer.

Tensor parallelism slices every expert across all GPUs, so each GPU computes a fraction of every expert for every token. Expert parallelism places whole experts on different GPUs and sends each token only to the experts it selected.

The better cut depends on expert size, tokens per batch, and the links between GPUs. Work the arithmetic for weights, collectives, and dispatch before committing a cluster to either.

Expert parallelism vs tensor parallelism in one layer

An MoE layer replaces one feed-forward network with many smaller experts and a router. Each token runs through only a few of them.

Mixtral 8x7B has 8 experts per layer and routes each token to 2. The model stores 47 billion parameters but uses about 13 billion per token.

The TensorRT-LLM expert parallelism guide defines both cuts.

In tensor parallel mode, each expert's weights are split evenly, so every rank holds partial weights of all experts.

In expert parallel mode, each rank holds the full weights of a subset of experts. A tensor parallel rank receives every token's hidden state. An expert parallel rank receives only the tokens routed to its experts.

Placing experts on separate accelerators goes back to GShard, which scaled a sparsely gated translation model past 600 billion parameters with automatic sharding.

Fanout's MoE routing guide explains how the router selects experts before any parallelism question appears.

Weight memory per GPU is roughly equal either way

Put Mixtral's experts on 8 GPUs. With tensor parallelism of degree 8, each GPU stores one eighth of all 8 experts in a layer. With expert parallelism of degree 8, each GPU stores one full expert per layer.

Both layouts hold one eighth of the layer's expert bytes per GPU. Memory fit therefore rarely picks the winner between the two for the expert weights themselves.

What changes is the shape of the compute and the traffic. One layout runs a sliver of every expert on every token. The other runs whole experts on a subset of tokens.

Attention, embeddings, and any shared experts sit outside this choice. Serving engines shard or replicate them separately, which is why real configurations mix several parallel modes.

Tensor parallelism runs dense collectives every MoE layer

Sharded layers must merge partial results. Fanout's tensor vs pipeline parallelism guide derives the ring all-reduce cost; here the relevant part is scale.

Take a hidden width of 7,168 in BF16 at 2 bytes per element. One token's hidden state is 14,336 bytes, about 14 KiB.

A ring all-reduce across 8 ranks moves about 2 times 7/8 of the payload per rank, near 25 KiB per rank per token for each collective. Every rank synchronizes on every token, whichever experts that token picked.

The collective is dense, but the matrix work behind it gets thin. DeepSeek-V3 has 256 routed experts per MoE layer, and each is deliberately small.

Slicing an already small expert 8 ways leaves each rank a sliver of a matrix multiplication. Small GEMMs waste GPU throughput, and the all-reduce tax stays regardless.

This is why fine-grained MoE designs avoid tensor parallelism inside expert layers. The DeepSeek-V3 report notes the model was trained without costly tensor parallelism at all.

Expert parallelism pays in all-to-all dispatch

Expert parallelism replaces the all-reduce with dispatch and combine. Each token's hidden state travels to the ranks holding its selected experts, and the outputs travel back.

With top-8 routing and a 14 KiB hidden state, one token can send up to 8 copies outward, near 112 KiB, and receive a similar volume back. The upper bound is roughly 224 KiB per token per MoE layer.

Routing policy caps this in practice. DeepSeek-V3 limits each token to at most 4 nodes, so cross-node traffic on InfiniBand stays bounded while NVLink handles the intra-node fanout.

The pattern is irregular, which is its main risk. A hot expert turns one rank into a queue while others idle.

The vLLM expert parallel deployment guide ships a load balancer, EPLB, that re-places experts under skew.

Redundant copies cost real memory, about 2.4 GB per extra DeepSeek-V3 expert.

Batch occupancy decides which pattern wins

Expert parallelism earns its dispatch cost only when each expert receives enough tokens to run an efficient batched matrix multiplication.

vLLM exposes this directly: its deepep_high_throughput backend targets prefill, where token counts are large, while deepep_low_latency targets decode, where each step carries few tokens per expert.

At low concurrency, most expert ranks sit idle each step. Tensor parallelism keeps all GPUs working on every token, so small deployments often measure better latency without expert parallelism.

One published measurement of a roughly 35B-parameter MoE on H100s found expert parallelism combined with tensor parallelism of degree 4 beat the no-EP baseline by about 33 percent in output throughput.

The same Jarvis Labs benchmark found expert parallelism with tensor parallelism of degree 2 consistently below its baseline.

Treat both results as one engine, one model, and one fabric, not a universal ranking.

Fanout's continuous batching guide covers why concurrency sets the token counts these kernels see in the first place.

How a production MoE deployment combines the modes

DeepSeek-V3's inference deployment uses different layouts per phase. Prefill runs on 4 nodes with 32 GPUs.

Prefill attention uses tensor parallelism of degree 4 with data parallelism of 8. The MoE layers use expert parallelism across all 32 GPUs.

Decode runs on 40 nodes with 320 GPUs. Expert parallelism spans all 320, one expert per GPU, with 64 GPUs reserved for redundant and shared experts.

Tensor parallelism still appears, but only inside attention, where the weights are large and dense. The expert layers stay whole and spread wide.

vLLM encodes the same composition rule: expert parallel size equals tensor parallel size times data parallel size, enabled with a single flag on top of an existing layout.

The phase split matters because prefill and decode want different all-to-all kernels. Fanout's disaggregated prefill and decode guide covers separating the phases onto different pools.

Decide from expert shape, batch, and fabric

First, count experts and their size. Hundreds of small experts argue for expert parallelism, because slicing them produces inefficient fragments. A few large experts tolerate tensor parallelism well.

Second, estimate tokens per expert per step from real concurrency. Divide the batch's routed token count by active experts; single-digit results mean expert ranks will starve during decode.

Third, map the fabric. All-to-all across slow links needs a routing cap or a hierarchical scheme before it is viable, while NVLink-class domains absorb either pattern.

Fourth, benchmark the hybrid, not just the extremes. TensorRT-LLM lets tensor and expert degrees multiply to the same GPU count, and the best split is workload-specific.

Measure time to first token, inter-token latency, and throughput at the same concurrency for each candidate.

The winning layout keeps expert GEMMs full and collectives on the fastest links, not the nicest diagram.