Model architecture glossary.

Plain-English explanations of the blocks you see in modern model architectures - the same explanations that appear when you hover nodes in hfviewer's interactive graphs, grounded in each model's real numbers.

Attention

QK-NormNormalizes queries and keys right before attention so their dot products can't blow up.
Multi-head latent attention (MLA)Compresses keys and values into one small latent vector per token - the cache stores the latent, not full K/V.
Sparse attention (top-k token selection)A cheap 'indexer' scores past tokens and full attention runs only over the top-scoring ones.
Sliding-window attentionEach token only attends to a fixed window of recent tokens, keeping cost and cache flat as context grows.
Linear attention (gated DeltaNet)Replaces the attention matrix with a fixed-size recurrent state - cost stays constant per token, no KV cache growth.
Mamba / state-space block (SSM)A recurrent state-space scan replaces attention: constant memory per token, linear time in sequence length.
Short convolution blockA gated, depthwise causal convolution mixes only a few neighboring tokens - a very cheap substitute for attention.
Grouped-query attention (GQA)Several query heads share one key/value head, shrinking the KV cache several-fold with little quality loss.
Self-attentionEvery token builds queries, keys and values, then gathers information from the other tokens that match its query.
Attention sinkA learned 'nowhere' slot lets a head cleanly attend to nothing instead of smearing weight over random tokens.
KV cacheGeneration reuses cached keys/values of all previous tokens instead of recomputing them - memory becomes the bottleneck.

Positions & context

Norms & blocks

Mixture of experts

Heads & prediction

Multimodal

Diffusion & generation

Audio

Every concept here links to real models where you can see the block in context - zoom, expand, and inspect the actual architecture.

Browse all 2,300+ model graphs →