Mixture of Experts
Think of a library where, instead of one librarian trying to know everything, there is a team of specialists. When you ask a question, a 'router' quickly directs you to the two or three experts best suited to answer it, while the rest of the team stays quiet. This allows the system to be incredibly smart and knowledgeable without needing to use all its brainpower for every single word it generates.
A sparse neural network architecture that decouples model capacity from computational cost. It utilizes a gating mechanism (router) to dynamically select a subset of feed-forward sub-networks (experts) for each input token. This enables the model to scale to massive parameter counts while maintaining a constant, efficient inference cost per token.
An architecture with many specialised sub-networks ('experts') and a router that activates only a few per token, raising total parameter count without a proportional increase in compute.
evolution
- 1991 · historyAdaptive Mixtures of Local Experts
Jacobs et al. introduced the concept of modular neural networks where a gating network selects specific experts for different input regions.
- 2017 · historyOutrageously Large Neural Networks
Shazeer et al. scaled the MoE architecture to billions of parameters using a sparsity-based gating mechanism for deep learning models.
- 2021 · historySwitch Transformer
Fedus et al. simplified the MoE routing mechanism, enabling the training of trillion-parameter models with significantly reduced computational costs.
- 2024 · historyMixture-of-Experts in Production LLMs
The release of models like Mixtral 8x7B demonstrated that MoE architectures could achieve state-of-the-art performance with efficient inference.