Mixture of experts

Shared expert

An always-on expert that every token passes through, alongside whatever the router picks.

First page of DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsIntroduced inJan 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDai et al. · arXiv 2401.06066 ↗
The shared expert runs for every tokenrouterexpert 1expert 2expert 3expert 4expert 5expert 6expert 7expert 8always on →shared expertrouted experts specialize; the shared one carries common knowledge

DeepSeek-style MoE adds one or more shared experts that process every token unconditionally, in parallel with the routed top-k experts. The shared expert captures common knowledge every token needs, freeing the routed experts to specialize harder. Which measurably improves quality at the same compute. Most 2025-generation MoE models (DeepSeek, GLM, Qwen-MoE, Nemotron) include one.

Adoption over time

Share of new models that have included a shared expert over time.

19%2022202320242025202619%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 13.1M · ♡ 357Open in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 12.6M · ♡ 563Open in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 5.4M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 5.2M · ♡ 14kOpen in visualizer ornith-ai/Ornith-1.0-35B architecture graphornith-ai/Ornith-1.0-35Btext-generation · ↓ 3.2M · ♡ 505Open in visualizer deepreinforce-ai/Ornith-1.0-35B architecture graphdeepreinforce-ai/Ornith-1.0-35Btext-generation · ↓ 3.2M · ♡ 505Open in visualizer baidu/Unlimited-OCR architecture graphbaidu/Unlimited-OCRimage-text-to-text · ↓ 3.1M · ♡ 4kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 3.1M · ♡ 4kOpen in visualizer moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 2.7M · ♡ 11kOpen in visualizer zai-org/GLM-5.2 architecture graphzai-org/GLM-5.2text-generation · ↓ 2.7M · ♡ 5kOpen in visualizer unsloth/Qwen3.6-35B-A3B-NVFP4 architecture graphunsloth/Qwen3.6-35B-A3B-NVFP4image-text-to-text · ↓ 2.6M · ♡ 113Open in visualizer Qwen/Qwen3.5-35B-A3B architecture graphQwen/Qwen3.5-35B-A3Bimage-text-to-text · ↓ 2.5M · ♡ 1kOpen in visualizer

Browse all 394 models with this in the catalog →

Related concepts

Grouped expert routing MoE router MoE experts

hfviewer renders the full architecture of 4,000+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →