Mixture of experts

MoE experts

A bank of parallel feed-forward networks; each token runs through only the few the router selected.

First page of Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerIntroduced inJan 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer et al. · arXiv 1701.06538 ↗
Mixture of experts: top-2 of 8routerexpert 1expert 2expert 3expert 4expert 5expert 6expert 7expert 8stored: 8 experts · used per token: 2

Each expert is an independent gated MLP. Because only the top-k experts execute per token, total parameter count and per-token compute decouple: a model can store hundreds of experts' worth of knowledge while running just a handful. Different experts end up specializing. By language, domain, or syntax. Although the specialization is learned, not assigned.

Adoption over time

Share of new models that have included MoE experts over time.

28%2022202320242025202628%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 11.7M · ♡ 1kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 11.3M · ♡ 534Open in visualizer deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 9.9M · ♡ 14kOpen in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 8.9M · ♡ 341Open in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 8.6M · ♡ 5kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 5.9M · ♡ 3kOpen in visualizer openai/gpt-oss-120b architecture graphopenai/gpt-oss-120btext-generation · ↓ 4.2M · ♡ 5kOpen in visualizer cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit architecture graphcyankiwi/gemma-4-26B-A4B-it-AWQ-4bitimage-text-to-text · ↓ 3.7M · ♡ 91Open in visualizer Qwen/Qwen3-30B-A3B architecture graphQwen/Qwen3-30B-A3Btext-generation · ↓ 3.1M · ♡ 916Open in visualizer deepreinforce-ai/Ornith-1.0-35B architecture graphdeepreinforce-ai/Ornith-1.0-35Btext-generation · ↓ 2.8M · ♡ 467Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 2.8M · ♡ 415Open in visualizer deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 2.7M · ♡ 2kOpen in visualizer

Browse all 394 models with this in the catalog →

Related concepts

Grouped expert routing MoE router Shared expert

hfviewer renders the full architecture of 3,100+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →