Attention

Self-attention

Every token builds queries, keys and values, then gathers information from the other tokens that match its query.

First page of Attention Is All You NeedIntroduced inJun 2017Attention Is All You NeedVaswani et al. · arXiv 1706.03762 ↗
Self-attention: every token asks, all others answerthe query compares with each earlier token · weights (bars) set the blendquery

Self-attention is the core transformer operation: each position emits a query that is compared against the keys of the other positions, and the resulting weights average their values. It is what lets the model relate any token to any other, at a cost that grows with the square of sequence length. Which is why so many of the surrounding tricks (GQA, sliding windows, sparse and linear attention) exist to tame it.

Adoption over time

Share of new models that have included self-attention over time.

25%2022202320242025202625%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google-bert/bert-base-uncased architecture graphgoogle-bert/bert-base-uncasedfill-mask · ↓ 119.5M · ♡ 3kOpen in visualizer cross-encoder/ms-marco-MiniLM-L6-v2 architecture graphcross-encoder/ms-marco-MiniLM-L6-v2text-ranking · ↓ 89.3M · ♡ 297Open in visualizer BAAI/bge-small-en-v1.5 architecture graphBAAI/bge-small-en-v1.5feature-extraction · ↓ 72.9M · ♡ 525Open in visualizer sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 architecture graphsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2sentence-similarity · ↓ 60.0M · ♡ 1kOpen in visualizer google/electra-base-discriminator architecture graphgoogle/electra-base-discriminator↓ 56.6M · ♡ 149Open in visualizer BAAI/bge-m3 architecture graphBAAI/bge-m3sentence-similarity · ↓ 35.2M · ♡ 3kOpen in visualizer google-t5/t5-small architecture graphgoogle-t5/t5-smalltranslation · ↓ 25.1M · ♡ 589Open in visualizer FacebookAI/xlm-roberta-base architecture graphFacebookAI/xlm-roberta-basefill-mask · ↓ 20.9M · ♡ 878Open in visualizer BAAI/bge-reranker-v2-m3 architecture graphBAAI/bge-reranker-v2-m3text-classification · ↓ 19.2M · ♡ 1kOpen in visualizer intfloat/multilingual-e5-small architecture graphintfloat/multilingual-e5-smallsentence-similarity · ↓ 15.6M · ♡ 380Open in visualizer openai-community/gpt2 architecture graphopenai-community/gpt2text-generation · ↓ 13.8M · ♡ 3kOpen in visualizer FacebookAI/roberta-large architecture graphFacebookAI/roberta-largefill-mask · ↓ 13.4M · ♡ 318Open in visualizer

Browse all 1049 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA)

hfviewer renders the full architecture of 3,100+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →