Architecture
aka MoE
Mixture of Experts
Architecture where a router activates only a few expert sub-networks per token, saving compute.
Definition
Mixture-of-experts replaces dense feed-forward layers with many parallel experts, of which a router selects 1â2 per token. This decouples total parameter count from active compute: huge models stay cheap to serve. Mixtral 8x22B and DeepSeek-V3 are popular open MoEs.
Common use cases
- Cheap large models
- Specialised routing
- Compute-efficient scaling
Related models
mistral
deepseek
dbrx