How MoE Inference Differs From Dense Models

A model can select a submodel of roughly 13 billion parameters per token and still require more than 90 GB for its weights. Buying memory from the smaller number leaves most of the model nowhere to live. Mixture-of-experts inference separates two budgets that were close together for the dense course model: the weights stored and the weights selected for a…