Attention

QK-Norm

Normalizes queries and keys right before attention so their dot products can’t blow up.

First page of Query-Key Normalization for TransformersIntroduced inOct 2020Query-Key Normalization for TransformersHenry et al. · arXiv 2010.04245 ↗
QK-Norm: normalize before comparingQ and K are normalized onto the unit circle · only the angle mattersQKq·k = 47.3 · saturatesq·k = 0.8 · stabledot products can’t blow up

QK-Norm applies RMSNorm to the query and key vectors just before they are compared. Without it, query-key dot products can grow so large during training that attention saturates onto a single token and destabilizes the loss. It became standard in recent models (Qwen3+, Gemma) because it lets them train at higher learning rates with almost no extra compute.

Adoption over time

Share of new models that have included QK-Norm ops over time.

57%2022202320242025202657%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 24.5M · ♡ 2kOpen in visualizer Comfy-Org/MiniMax-H3 architecture graphComfy-Org/MiniMax-H3↓ 18.3M · ♡ 2kOpen in visualizer Qwen/Qwen3-8B architecture graphQwen/Qwen3-8Btext-generation · ↓ 14.6M · ♡ 1kOpen in visualizer Qwen/Qwen3.6-35B-A3B-FP8 architecture graphQwen/Qwen3.6-35B-A3B-FP8image-text-to-text · ↓ 13.4M · ♡ 358Open in visualizer nvidia/Qwen3.6-35B-A3B-NVFP4 architecture graphnvidia/Qwen3.6-35B-A3B-NVFP4text-generation · ↓ 12.2M · ♡ 569Open in visualizer Qwen/Qwen3.6-27B-FP8 architecture graphQwen/Qwen3.6-27B-FP8image-text-to-text · ↓ 9.0M · ♡ 347Open in visualizer google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 8.8M · ♡ 1kOpen in visualizer google/gemma-4-31B-it architecture graphgoogle/gemma-4-31B-itimage-text-to-text · ↓ 8.8M · ♡ 4kOpen in visualizer Qwen/Qwen3.5-4B architecture graphQwen/Qwen3.5-4Bimage-text-to-text · ↓ 7.8M · ♡ 848Open in visualizer Qwen/Qwen3.6-27B architecture graphQwen/Qwen3.6-27Bimage-text-to-text · ↓ 6.3M · ♡ 2kOpen in visualizer Qwen/Qwen3-VL-8B-Instruct architecture graphQwen/Qwen3-VL-8B-Instructimage-text-to-text · ↓ 6.1M · ♡ 1kOpen in visualizer Qwen/Qwen3.6-35B-A3B architecture graphQwen/Qwen3.6-35B-A3Bimage-text-to-text · ↓ 5.4M · ♡ 3kOpen in visualizer

Browse all 1251 models with this in the catalog →

Related concepts

Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 4,000+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →