Attention

Sliding-window attention

Each token only attends to a fixed window of recent tokens, keeping cost and cache flat as context grows.

First page of Longformer: The Long-Document TransformerIntroduced inApr 2020Longformer: The Long-Document TransformerBeltagy et al. · arXiv 2004.05150 ↗
Full attentionSliding windowevery token sees all before iteach token sees the last 4hybrid stacks alternate:SSSFSSSF

Rather than looking at the entire history, a sliding-window layer attends only to the last N tokens. Models like Gemma and gpt-oss interleave many sliding layers with occasional full-attention layers: the sliding layers handle local structure cheaply while the periodic full layers carry long-range information. This hybrid keeps the KV cache small without giving up long-context ability.

Adoption over time

Share of new models that have included sliding-window attention over time.

16%2022202320242025202616%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

google/gemma-4-31B-it architecture graphgoogle/gemma-4-31B-itimage-text-to-text · ↓ 11.9M · ♡ 3kOpen in visualizer google/gemma-4-26B-A4B-it architecture graphgoogle/gemma-4-26B-A4B-itimage-text-to-text · ↓ 11.7M · ♡ 1kOpen in visualizer pyannote/speaker-diarization-3.1 architecture graphpyannote/speaker-diarization-3.1automatic-speech-recognition · ↓ 8.9M · ♡ 3kOpen in visualizer openai/gpt-oss-20b architecture graphopenai/gpt-oss-20btext-generation · ↓ 8.6M · ♡ 5kOpen in visualizer answerdotai/ModernBERT-base architecture graphanswerdotai/ModernBERT-basefill-mask · ↓ 6.4M · ♡ 1kOpen in visualizer google/gemma-4-E4B-it architecture graphgoogle/gemma-4-E4B-itany-to-any · ↓ 5.8M · ♡ 1kOpen in visualizer openai/gpt-oss-120b architecture graphopenai/gpt-oss-120btext-generation · ↓ 4.2M · ♡ 5kOpen in visualizer google/gemma-4-E2B-it architecture graphgoogle/gemma-4-E2B-itany-to-any · ↓ 4.2M · ♡ 870Open in visualizer ibm-granite/granite-embedding-small-english-r2 architecture graphibm-granite/granite-embedding-small-english-r2feature-extraction · ↓ 3.8M · ♡ 75Open in visualizer cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit architecture graphcyankiwi/gemma-4-26B-A4B-it-AWQ-4bitimage-text-to-text · ↓ 3.7M · ♡ 91Open in visualizer google/gemma-4-12B-it architecture graphgoogle/gemma-4-12B-itany-to-any · ↓ 3.1M · ♡ 1kOpen in visualizer nvidia/Gemma-4-31B-IT-NVFP4 architecture graphnvidia/Gemma-4-31B-IT-NVFP4text-generation · ↓ 2.8M · ♡ 547Open in visualizer

Browse all 321 models with this in the catalog →

Related concepts

QK-Norm Multi-head latent attention (MLA) Sparse attention (top-k token selection) Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 3,100+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →