Attention

Short convolution block

A gated, depthwise causal convolution mixes only a few neighboring tokens. A very cheap substitute for attention.

First page of Hungry Hungry Hippos: Towards Language Modeling with State Space ModelsGoes back toDec 2022Hungry Hungry Hippos: Towards Language Modeling with State Space ModelsFu et al. · arXiv 2212.14052 ↗
Short conv block: mix only nearby tokensa gated depthwise causal conv · each output sees just its few neighboursinputsoutputs3-wide causal window

Some hybrid models (LFM2, Nemotron-H) replace most attention layers with short causal convolutions: each position mixes information from just the last few tokens through a depthwise conv, with multiplicative gates deciding what passes through. It is dramatically cheaper than attention and surprisingly capable for local patterns; the few remaining attention layers supply global context.

Adoption over time

Share of new models that have included short convolution blocks over time.

3%202220232024202520263%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 2.7M · ♡ 11kOpen in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 1.5M · ♡ 429Open in visualizer Qwen/Qwen3-Omni-30B-A3B-Instruct architecture graphQwen/Qwen3-Omni-30B-A3B-Instructany-to-any · ↓ 995k · ♡ 987Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16text-generation · ↓ 965k · ♡ 813Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16text-generation · ↓ 963k · ♡ 423Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8text-generation · ↓ 649k · ♡ 357Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4text-generation · ↓ 533k · ♡ 174Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-4B-BF16text-generation · ↓ 528k · ♡ 114Open in visualizer nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4text-generation · ↓ 442k · ♡ 128Open in visualizer nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16text-generation · ↓ 426k · ♡ 329Open in visualizer nvidia/NVIDIA-Nemotron-Nano-9B-v2 architecture graphnvidia/NVIDIA-Nemotron-Nano-9B-v2text-generation · ↓ 390k · ♡ 513Open in visualizer nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4text-generation · ↓ 330k · ♡ 311Open in visualizer

Browse all 64 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 4,000+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →