Mixture of experts

MoE router

Scores all experts per token and activates only the top-k. Most of the model’s weights stay idle each step.

First page of Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerIntroduced inJan 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer et al. · arXiv 1701.06538 ↗
The router picks 2 of 8 experts per tokenrouterexpert 1expert 2expert 3expert 4expert 5expert 6expert 7expert 8different tokens light up different experts · specialization is learned

In a mixture-of-experts layer the router is a small linear layer that scores every expert for the current token; only the top-k scorers actually run, and their outputs are combined with the router’s weights. This is how models grow to hundreds of billions of parameters while spending the compute of a much smaller dense model per token. Routers are trained with auxiliary objectives (or bias tricks) to keep expert load balanced.

Adoption over time

Share of new models that have included an MoE router over time.

37%2022202320242025202637%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 13.6M · ♡ 1kOpen in visualizer deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 8.7M · ♡ 13kOpen in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 8.7M · ♡ 483Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 7.6M · ♡ 318Open in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 7.4M · ♡ 5kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 6.2M · ♡ 2kOpen in visualizer cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit architecture graphcyankiwi/gemma-4-26B-A4B-it-AWQ-4bitimage-text-to-text · ↓ 5.1M · ♡ 87Open in visualizer openai/gpt-oss-120b architecture graphopenai/gpt-oss-120btext-generation · ↓ 4.5M · ♡ 5kOpen in visualizer deepseek-ai/DeepSeek-OCR-2 architecture graphdeepseek-ai/DeepSeek-OCR-2image-text-to-text · ↓ 3.1M · ♡ 1kOpen in visualizer zai-org/GLM-5.2-FP8 architecture graphzai-org/GLM-5.2-FP8text-generation · ↓ 3.0M · ♡ 223Open in visualizer deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 3.0M · ♡ 2kOpen in visualizer Qwen/Qwen3-Coder-Next-FP8 architecture graphQwen/Qwen3-Coder-Next-FP8text-generation · ↓ 2.9M · ♡ 161Open in visualizer

Browse all 286 models with this in the catalog →

Related concepts

Grouped expert routing Shared expert MoE experts

hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →