Logit softcapping
Squashes attention or output logits through a scaled tanh so no single value can run away.
Popularized inJul 2024Gemma 2: Improving Open Language Models at a Practical Size
Softcapping bounds logits to a fixed range by passing them through tanh scaled by a cap value. Gemma-2 popularized it for both attention scores and final output logits as a training stabilizer. Extreme logits otherwise cause saturation and loss spikes. Newer models often replace it with QK-Norm, but you will still find it in several current architectures.
Adoption over time
Share of new models that have included logit softcapping over time.
See it in real models
Open any of these on hfviewer to find this block in the interactive architecture graph.
meta-models/Muse-Glimmer-30Bimage-text-to-text · ↓ 282k · ♡ 2kOpen in visualizer
RedHatAI/Muse-Glimmer-30B-FP8-blockimage-text-to-text · ↓ 46k · ♡ 12Open in visualizer
nvidia/Muse-Glimmer-30B-NVFP4image-text-to-text · ↓ 25k · ♡ 15Open in visualizer
RadixArk/Muse-Glimmer-NVFP4image-text-to-text · ↓ 23k · ♡ 9Open in visualizer
RedHatAI/Muse-Glimmer-30B-INT4image-text-to-text · ↓ 10k · ♡ 8Open in visualizer
RedHatAI/Muse-Glimmer-30B-W4A16image-text-to-text · ↓ 10k · ♡ 8Open in visualizer
unsloth/Muse-Glimmer-30B-unsloth-bnb-4bitimage-text-to-text · ↓ 8k · ♡ 14Open in visualizer
darkc0de/Muse-Glimmer-30B-hereticimage-text-to-text · ↓ 5k · ♡ 15Open in visualizer
RedHatAI/Muse-Glimmer-30B-NVFP4image-text-to-text · ↓ 4k · ♡ 18Open in visualizer
unsloth/Muse-Glimmer-30Bimage-text-to-text · ↓ 4k · ♡ 14Open in visualizer
RedHatAI/Muse-Glimmer-30Bimage-text-to-text · ↓ 3kOpen in visualizer
dmis-lab/Gemma4-12B-CVRRvisual-question-answering · ↓ 2kOpen in visualizer
View 19 catalog matches for Logit softcapping →
Related concepts
hfviewer renders the full architecture of 6,700+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.
Browse all model graphs →