A learned 'nowhere' slot lets a head cleanly attend to nothing instead of smearing weight over random tokens.
Softmax forces attention weights to sum to one, so a head that has nothing relevant to look at must still put its weight somewhere - often degrading quality. An attention sink adds a learned logit (or a pinned first token) that absorbs that leftover probability. gpt-oss bakes per-head sink parameters into every attention layer; the same idea is why streaming-inference tricks keep the first tokens around.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →