Every token builds queries, keys and values, then gathers information from the other tokens that match its query.
Self-attention is the core transformer operation: each position emits a query that is compared against the keys of the other positions, and the resulting weights average their values. It is what lets the model relate any token to any other, at a cost that grows with the square of sequence length - which is why so many of the surrounding tricks (GQA, sliding windows, sparse and linear attention) exist to tame it.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →