Rescales activations by their RMS magnitude - like LayerNorm without mean-centering, and cheaper.
RMSNorm divides each activation vector by its root-mean-square and multiplies by a learned scale, skipping the mean subtraction and bias that LayerNorm carries. It stabilizes training just as well in practice while being simpler and faster, which is why almost every modern LLM (Llama onward) uses it, typically placed before attention and before the MLP in each block ('pre-norm').
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →