Attention

Short convolution block

A gated, depthwise causal convolution mixes only a few neighboring tokens. A very cheap substitute for attention.

First page of Hungry Hungry Hippos: Towards Language Modeling with State Space ModelsGoes back toDec 2022Hungry Hungry Hippos: Towards Language Modeling with State Space ModelsFu et al. · arXiv 2212.14052 ↗
Short conv block: mix only nearby tokensa gated depthwise causal conv · each output sees just its few neighboursinputsoutputs3-wide causal window

Some hybrid models (LFM2, Nemotron-H) replace most attention layers with short causal convolutions: each position mixes information from just the last few tokens through a depthwise conv, with multiplicative gates deciding what passes through. It is dramatically cheaper than attention and surprisingly capable for local patterns; the few remaining attention layers supply global context.

Adoption over time

Share of new models that have included short convolution blocks over time.

3%202220232024202520263%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 2.8M · ♡ 415Open in visualizer Qwen/Qwen3-Omni-30B-A3B-Instruct architecture graphQwen/Qwen3-Omni-30B-A3B-Instructany-to-any · ↓ 1.4M · ♡ 969Open in visualizer moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 1.1M · ♡ 10kOpen in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16text-generation · ↓ 931k · ♡ 805Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16text-generation · ↓ 899k · ♡ 414Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4text-generation · ↓ 765k · ♡ 174Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-4B-BF16text-generation · ↓ 739k · ♡ 105Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8text-generation · ↓ 613k · ♡ 355Open in visualizer nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16text-generation · ↓ 469k · ♡ 303Open in visualizer nvidia/NVIDIA-Nemotron-Nano-9B-v2 architecture graphnvidia/NVIDIA-Nemotron-Nano-9B-v2text-generation · ↓ 313k · ♡ 506Open in visualizer nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4text-generation · ↓ 233k · ♡ 279Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8text-generation · ↓ 208k · ♡ 271Open in visualizer

Browse all 37 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 3,100+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →