Attention

Mamba / state-space block (SSM)

A recurrent state-space scan replaces attention: constant memory per token, linear time in sequence length.

First page of Mamba: Linear-Time Sequence Modeling with Selective State SpacesIntroduced inDec 2023Mamba: Linear-Time Sequence Modeling with Selective State SpacesGu & Dao · arXiv 2312.00752 ↗
Attentionthe KV cache grows with every tokenSelective state (Mamba)conv, then a gated scan rewrites SScost per token growscost stays constant

State-space blocks (Mamba-2’s SSD scan here) carry information through a fixed-size hidden state that is updated token by token, instead of comparing every token against every other one. That makes both compute and memory linear in sequence length. Hybrid models such as Nemotron-H use mostly Mamba blocks with a few attention layers mixed in for tasks that need exact token recall.

Adoption over time

Share of new models that have included Mamba/SSM blocks over time.

2%202220232024202520262%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 architecture graphnvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4any-to-any · ↓ 1.5M · ♡ 178Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4text-generation · ↓ 1.5M · ♡ 429Open in visualizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 architecture graphnvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8any-to-any · ↓ 969k · ♡ 61Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16text-generation · ↓ 965k · ♡ 813Open in visualizer nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16text-generation · ↓ 963k · ♡ 423Open in visualizer ai21labs/Jamba-tiny-dev architecture graphai21labs/Jamba-tiny-dev↓ 750k · ♡ 14Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8text-generation · ↓ 649k · ♡ 357Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4text-generation · ↓ 533k · ♡ 174Open in visualizer nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 architecture graphnvidia/NVIDIA-Nemotron-3-Nano-4B-BF16text-generation · ↓ 528k · ♡ 114Open in visualizer microsoft/Mage-VL architecture graphmicrosoft/Mage-VLimage-text-to-text · ↓ 494k · ♡ 366Open in visualizer state-spaces/mamba-130m-hf architecture graphstate-spaces/mamba-130m-hftext-generation · ↓ 483k · ♡ 72Open in visualizer nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 architecture graphnvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4text-generation · ↓ 442k · ♡ 128Open in visualizer

Browse all 61 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 4,000+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →