Replaces the attention matrix with a fixed-size recurrent state - cost stays constant per token, no KV cache growth.
Linear attention keeps a compact running state that is updated once per token (here with the gated delta rule: the state is selectively decayed and rewritten), so generation cost does not grow with context length and there is no per-token KV cache. Qwen3.5-style hybrids use linear attention in most layers and keep periodic full-attention layers for precise long-range recall - the best of both worlds for long contexts.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →