This path deliberately skips position encoding - the model relies on causal masking alone to sense order.
Some recent models apply RoPE only to part of the head dimensions or only in some layers, leaving the rest position-free ('NoPE'). Because causal masking already gives a weak sense of order, dropping explicit positions in places can improve length generalization: the model overfits less to the exact positions it saw in training. You will see graphs split query/key paths into an explicitly rotated part and a NoPE part.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →