Mixture of experts

Scale parameters without scaling compute: replace each MLP with many 'expert' MLPs and a router that activates only a couple per token. A trillion-parameter MoE can run at the cost of a much smaller dense model — the trick behind Mixtral, DeepSeek-V3, and most frontier models today. The catch: routing balance and memory footprint.