Mixture of experts

MoE experts

A bank of parallel feed-forward networks; each token runs through only the few the router selected.

First page of Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerIntroduced inJan 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer et al. · arXiv 1701.06538 ↗
Mixture of experts: top-2 of 8routerexpert 1expert 2expert 3expert 4expert 5expert 6expert 7expert 8stored: 8 experts · used per token: 2

Each expert is an independent gated MLP. Because only the top-k experts execute per token, total parameter count and per-token compute decouple: a model can store hundreds of experts' worth of knowledge while running just a handful. Different experts end up specializing. By language, domain, or syntax. Although the specialization is learned, not assigned.

Adoption over time

Share of new models that have included MoE experts over time.

19%2022202320242025202619%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 13.1M · ♡ 2kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 8.0M · ♡ 628Open in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 6.7M · ♡ 5kOpen in visualizer zai-org/GLM-5.3-Flash architecture graphzai-org/GLM-5.3-Flashimage-text-to-text · ↓ 5.4M · ♡ 3kOpen in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 5.3M · ♡ 412Open in visualizer openai/gpt-oss-120b architecture graphopenai/gpt-oss-120btext-generation · ↓ 5.0M · ♡ 5kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 4.5M · ♡ 4kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 3.4M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-V3.2 architecture graphdeepseek-ai/DeepSeek-V3.2text-generation · ↓ 2.5M · ♡ 1kOpen in visualizer deepseek-ai/DeepSeek-OCR architecture graphdeepseek-ai/DeepSeek-OCRimage-text-to-text · ↓ 2.3M · ♡ 3kOpen in visualizer moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 2.0M · ♡ 11kOpen in visualizer ornith-ai/Ornith-1.0-35B architecture graphornith-ai/Ornith-1.0-35Btext-generation · ↓ 1.9M · ♡ 506Open in visualizer

View 840 catalog matches for MoE experts →

Related concepts

Grouped expert routing MoE router Shared expert

hfviewer renders the full architecture of 6,800+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →