The model's repeating unit: attention mixes information between tokens, an MLP transforms each token, with norms and residuals holding it together.
Nearly every modern model is a stack of identical blocks like this one. Each block first lets tokens exchange information through some variant of attention, then transforms every token independently through a (usually gated) MLP, with a normalization layer before each part and residual connections around them. Expand the block in the graph to see the exact wiring this model uses inside.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →