Norms & blocks

Gated MLP (SwiGLU)

The FFN splits into a ‘gate’ and an ‘up’ projection multiplied together. That’s why you see three Linears and a Mul.

First page of GLU Variants Improve TransformerIntroduced inFeb 2020GLU Variants Improve TransformerShazeer · arXiv 2002.05202 ↗
Gated MLP (SwiGLU): the gate opens and closes the flowgate path applies SiLU, then multiplies with up_proj before down_projgate_projup_projdown_projSiLU×silu(gate_proj(x)) × up_proj(x) flows through down_proj

Modern transformer MLPs compute silu(gate_proj(x)) × up_proj(x), then project back down with down_proj. A gated linear unit (SwiGLU when the activation is SiLU). The elementwise multiply lets the network dynamically decide which features to pass, and it consistently beats the classic two-layer ReLU MLP at equal parameter cost. This is why decoder-block MLPs show three Linear layers and a multiply instead of two Linears.

Adoption over time

Share of new models that have included a gated MLP (SwiGLU) over time.

80%2022202320242025202680%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 24.5M · ♡ 2kOpen in visualizer Comfy-Org/MiniMax-H3 architecture graphComfy-Org/MiniMax-H3↓ 18.3M · ♡ 2kOpen in visualizer Qwen/Qwen3-8B architecture graphQwen/Qwen3-8Btext-generation · ↓ 14.6M · ♡ 1kOpen in visualizer openai-community/gpt2 architecture graphopenai-community/gpt2text-generation · ↓ 14.3M · ♡ 3kOpen in visualizer Qwen/Qwen3.5-9B architecture graphQwen/Qwen3.5-9Bimage-text-to-text · ↓ 13.6M · ♡ 2kOpen in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 13.4M · ♡ 358Open in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 12.2M · ♡ 569Open in visualizer Qwen/Qwen2.5-7B-Instruct architecture graphQwen/Qwen2.5-7B-Instructtext-generation · ↓ 11.4M · ♡ 2kOpen in visualizer Qwen/Qwen3.6-27B-FP8 architecture graphQwen/Qwen3.6-27B-FP8image-text-to-text · ↓ 9.0M · ♡ 347Open in visualizer google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 8.8M · ♡ 1kOpen in visualizer Qwen/Qwen2.5-VL-7B-Instruct architecture graphQwen/Qwen2.5-VL-7B-Instructimage-text-to-text · ↓ 8.8M · ♡ 2kOpen in visualizer google/gemma-4-31B-it architecture graphgoogle/gemma-4-31B-itimage-text-to-text · ↓ 8.8M · ♡ 4kOpen in visualizer

Browse all 2636 models with this in the catalog →

Related concepts

RMSNorm Attention residuals (AttnRes) Residual (skip) connection Logit softcapping Hyper-connections (mHC) Transformer block

hfviewer renders the full architecture of 4,000+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →