Each token only attends to a fixed window of recent tokens, keeping cost and cache flat as context grows.
Rather than looking at the entire history, a sliding-window layer attends only to the last N tokens. Models like Gemma and gpt-oss interleave many sliding layers with occasional full-attention layers: the sliding layers handle local structure cheaply while the periodic full layers carry long-range information. This hybrid keeps the KV cache small without giving up long-context ability.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →