Mixture of experts

Shared expert

An always-on expert that every token passes through, alongside whatever the router picks.

First page of DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsIntroduced inJan 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDai et al. · arXiv 2401.06066 ↗
The shared expert runs for every tokenrouterexpert 1expert 2expert 3expert 4expert 5expert 6expert 7expert 8always on →shared expertrouted experts specialize; the shared one carries common knowledge

DeepSeek-style MoE adds one or more shared experts that process every token unconditionally, in parallel with the routed top-k experts. The shared expert captures common knowledge every token needs, freeing the routed experts to specialize harder. Which measurably improves quality at the same compute. Most 2025-generation MoE models (DeepSeek, GLM, Qwen-MoE, Nemotron) include one.

Adoption over time

Share of new models that have included a shared expert over time.

13%2022202320242025202613%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 8.0M · ♡ 628Open in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 5.3M · ♡ 412Open in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 4.5M · ♡ 4kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 3.4M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-V3.2 architecture graphdeepseek-ai/DeepSeek-V3.2text-generation · ↓ 2.5M · ♡ 1kOpen in visualizer deepseek-ai/DeepSeek-OCR architecture graphdeepseek-ai/DeepSeek-OCRimage-text-to-text · ↓ 2.3M · ♡ 3kOpen in visualizer moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 2.0M · ♡ 11kOpen in visualizer ornith-ai/Ornith-1.0-35B architecture graphornith-ai/Ornith-1.0-35Btext-generation · ↓ 1.9M · ♡ 506Open in visualizer deepreinforce-ai/Ornith-1.0-35B architecture graphdeepreinforce-ai/Ornith-1.0-35Btext-generation · ↓ 1.9M · ♡ 506Open in visualizer zai-org/GLM-4.7-Flash architecture graphzai-org/GLM-4.7-Flashtext-generation · ↓ 1.9M · ♡ 2kOpen in visualizer Qwen/Qwen3.5-35B-A3B architecture graphQwen/Qwen3.5-35B-A3Bimage-text-to-text · ↓ 1.8M · ♡ 2kOpen in visualizer Qwen/Qwen3.5-35B-A3B-FP8 architecture graphQwen/Qwen3.5-35B-A3B-FP8image-text-to-text · ↓ 1.6M · ♡ 156Open in visualizer

View 567 catalog matches for Shared expert →

Related concepts

Grouped expert routing MoE router MoE experts

hfviewer renders the full architecture of 6,700+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →