Rescales RoPE frequencies so a model trained on short contexts can operate on much longer ones.
YaRN and related RoPE-scaling schemes stretch the rotation frequencies of rotary embeddings so that positions beyond the training length still land in a range the model understands, usually with a temperature correction on attention. It is the standard way models advertise 128k+ contexts after being pretrained on much shorter windows.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →