Normalizes queries and keys right before attention so their dot products can't blow up.
QK-Norm applies RMSNorm to the query and key vectors just before they are compared. Without it, query-key dot products can grow so large during training that attention saturates onto a single token and destabilizes the loss. It became standard in recent models (Qwen3+, Gemma) because it lets them train at higher learning rates with almost no extra compute.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →