Squashes attention or output logits through a scaled tanh so no single value can run away.
Softcapping bounds logits to a fixed range by passing them through tanh scaled by a cap value. Gemma-2 popularized it for both attention scores and final output logits as a training stabilizer - extreme logits otherwise cause saturation and loss spikes. Newer models often replace it with QK-Norm, but you will still find it in several current architectures.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →