Encodes token positions by rotating query/key vectors - relative distances fall out of the math for free.
RoPE rotates each query and key vector by an angle proportional to its position, so the dot product between two tokens depends only on their relative distance. It needs no learned position table, extrapolates better than absolute embeddings, and is the de-facto standard in modern LLMs. Long-context variants rescale the rotation frequencies (see YaRN) to stretch the same mechanism to million-token windows.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →