--- title: "METRO AllGather vs AllToAll for MoE decode" description: "Compare METRO AllGather with conventional AllToAll using payload bytes, collective launch cost, activated experts, and decode-batch size." canonical_url: "https://fanout.sh/blog/metro-allgather-vs-alltoall-moe-decode" md_url: "https://fanout.sh/blog/metro-allgather-vs-alltoall-moe-decode.md" last_updated: "2026-10-07" access: "public" --- # METRO AllGather vs AllToAll for MoE decode Compare METRO AllGather with conventional AllToAll using payload bytes, collective launch cost, activated experts, and decode-batch size. - Author: Suraj Gaud - Published: 2026-10-07 - Track: Inference engineering - Access: Public - Tags: METRO, MoE, AllGather, AllToAll, expert parallelism, decode, NVLink, vLLM METRO AllGather vs AllToAll for MoE decode looks like an easy bandwidth verdict. AllGather copies every rank's tokens everywhere, while AllToAll sends each token toward its selected expert. METRO accepts the larger dispatch because small decode batches are often dominated by collective launch latency, not payload transmission. The global token view then enables a routing decision that loads fewer expert weights. The comparison has three terms: bytes moved, fixed collective cost, and expert-weight traffic after routing. Counting only the first term misses the reason METRO can win. ## METRO AllGather vs AllToAll changes the routing information Conventional expert parallelism computes top-k expert choices on each rank and dispatches selected token activations with AllToAll. Each rank sees its local choices. A load balancer can spread tokens across expert replicas, but it does not begin with one global top-k count for every expert. The[METRO paper](https://arxiv.org/html/2512.09277)replaces the dispatch before top-k with AllGather. Every rank receives the full decode token set and computes the same global expert demand vector. METRO then assigns each active expert to one available replica on the GPU with the fewest already activated experts. Expert outputs still return through an AllToAll combine. The collectives do different jobs. The larger initial payload buys the global information needed to minimize the maximum activated experts per GPU. Fanout's[activated experts comparison](/blog/eplb-activated-experts-vs-token-balance)explains why that objective differs from balancing token counts in a memory-bound decode batch. If a replica receives one token, its weights must still be available to the compute path. Spreading a small number of tokens across extra replicas can add more HBM traffic than it saves in token balance. ## Work the 8-GPU payload example The paper evaluates a representative case with 32 decode tokens per GPU, 8 GPUs, and fp16 activations. Its conventional AllToAll dispatch sends 256 KB per GPU. METRO's AllGather sends 2 MB per GPU, an 8 times larger payload. At 600 GB/s, 256 KB takes about 0.4 microseconds of ideal link time. The 2 MB AllGather payload takes about 3 microseconds. The byte difference is therefore about 1.75 MB per GPU, or roughly 2.9 microseconds at the stated link bandwidth. The paper compares those transfer times with tens to about 100 microseconds of fixed NCCL collective launch cost in this regime. Once fixed latency dominates both operations, multiplying the payload by eight does not multiply the observed collective time by eight. The[NCCL collective definitions](https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2293/user-guide/docs/usage/collectives.html)clarify the buffer shapes. AllGather takes N values from each of k ranks and produces k times N values on every rank. AllToAll splits each rank's input into k destination chunks and exchanges different chunks with every peer. These definitions describe logical volume, not the complete latency model. Protocol, topology, implementation, synchronization, and launch overhead determine the measured time. ## Add the expert-weight traffic term Decode is commonly memory-bound because a small batch provides little reuse while expert weights must move from HBM toward the compute units. In the paper's analytical model, activation traffic was less than 0.6 percent of expert-weight traffic even for a 1,000-token decode batch. That ratio explains why a larger activation collective can be a good trade. A few extra megabytes on NVLink may enable the router to avoid loading one or more large expert replicas from HBM. The paper's toy example uses 16 tokens across 8 GPUs. Perfect token balance sends two tokens to each GPU but activates two expert replicas per GPU. A routing that groups each expert's tokens on one replica activates one expert per GPU in the example. In the idealized memory-bound model, that halves the expert-weight loading term. METRO does not solve the expensive optimal assignment. It greedily assigns each active expert to the candidate GPU with the lowest activated-expert count. The measured routing result was within 10.9 percent of the optimal maximum expert count and up to 42.3 percent below the EPLB token-routing result. Fanout's[batch-size and expert-parallel guide](/blog/moe-balance-batch-size-vs-expert-parallel-size)covers the other side of the trade: wider expert parallelism also reduces tokens per local expert. METRO's objective matters most after the batch is small enough that expert GEMMs are memory-bound. In a compute-bound prefill, token balance can remain the better objective. ## Know what the paper actually measured The real-system experiment used Qwen3-30B on 8 A100 40 GB GPUs connected by 600 GB/s NVLink. It capped decode at 32 tokens per GPU, used full expert parallelism, and kept prefill and decode on the same system. Against EPLB routing at 1.5 times expert replication, METRO improved total token throughput by up to 15.9 percent and reduced decode latency by up to 12.9 percent in that setup. The larger-model results used a proprietary simulator for Qwen3-235B on 8 B200 GPUs and DeepSeek-V3 on 16 B200 GPUs. Those simulations reported up to 21.8 percent lower decode latency and 21.0 percent higher total token throughput at 1.5 times replication. The headline 4.11 times decode-throughput gain comes from a Pareto comparison at a fixed latency target. METRO's lower TPOT allowed a 4 times larger batch than EPLB at that point. That is not a claim that swapping AllToAll for AllGather makes one fixed configuration 4.11 times faster. It includes choosing a different batch size while holding the decode SLO constant. The paper also leaves prefill on EPLB token routing. METRO is applied to the memory-bound decode phase rather than presented as one routing rule for every phase. ## Find the crossover instead of copying 2 MB The 2 MB example is evidence for one topology and batch shape, not a universal budget. Build a measurement sweep over decode tokens per rank, expert-parallel degree, hidden width, activation dtype, and interconnect domain. For each point, record dispatch bytes, combine bytes, collective duration, router duration, maximum activated experts, expert-layer HBM bytes, and end-to-end TPOT. The first useful crossover asks when extra AllGather transmission time becomes comparable to fixed collective latency. If fixed overhead is 50 microseconds on a 600 GB/s path, the corresponding ideal payload is about 30 MB. Protocol efficiency and contention will lower the practical crossover. The second crossover asks when the decode batch becomes compute-bound. As tokens per active expert rise, the marginal cost of activating another replica can fall because its weights are reused across more work. The third crossover is topological. An 8-GPU island with uniform NVLink is not equivalent to an expert-parallel group that crosses nodes or slower links. AWS Neuron's[MoE architecture guide](https://awsdocs-neuron.readthedocs-hosted.com/en/v2.28.0/libraries/nxd-inference/developer_guides/moe-arch-deep-dive.html)describes the conventional concern from the other direction. Its current AllGather EP path sends all tokens to all ranks, while a planned AllToAll-v path would send only tokens assigned to each rank's experts. That guidance is not inconsistent with METRO. AllGather is wasteful when global visibility does not improve the downstream objective enough to repay its bytes. ## Distinguish METRO from backend names Serving stacks use terms such as AllGather, AllToAll, gather-reduce-scatter, and AllToAll-v for several expert dispatch implementations. An environment variable containing `allgather` does not imply that the runtime implements METRO's global top-k routing objective. AMD's[vLLM MoE playbook](https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html)shows an `allgather_reducescatter` backend in reproducible deployment commands. That backend choice describes a communication path. METRO also needs the global top-k vector and its activated-expert minimization kernel. Likewise, ordinary AllToAll and variable-sized AllToAll-v are not interchangeable. Padding, token imbalance, and destination counts can make physical bytes differ from a simple equal-chunk estimate. Confirm the runtime path with traces. Capture collective names, buffer sizes, rank group, start and end timestamps, and the chosen expert replica for each global expert count. Without the routing assignment, lower collective time can hide higher expert-weight traffic. Without collective traces, fewer activated experts can hide a network regression. ## Use a deployment decision rule Start with the production SLO rather than the paper's maximum throughput point. At each acceptable TPOT, compare the highest stable throughput for conventional routing and METRO-like routing on the same model, replication map, and hardware. Reject configurations that improve median TPOT but lose p95 under mixed prefill and decode. Collective contention can change when both phases share the interconnect. Keep output quality and expert selections fixed. METRO changes which replica serves an already selected expert, not which experts the model's top-k gate chooses. For strict low-latency targets, the paper found that batches of 64 tokens or fewer could move out of the memory-bound regime and become network-latency dominated. Its Pareto frontier then preferred more tensor parallelism, where expert load balancing was unnecessary. That is the practical stop condition for the METRO comparison. The decision is not AllGather or AllToAll in isolation. Use AllGather when global routing knowledge removes enough expert-weight traffic to repay the larger activation exchange at the required batch and topology. Measure all three terms. Bytes identify the network cost, collective traces identify fixed latency, and activated experts identify the HBM work the routing decision avoided. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md