A cheap 'indexer' scores past tokens and full attention runs only over the top-scoring ones.
Instead of attending to every previous token, a lightweight indexer ranks the context and selects a fixed budget of the most relevant tokens; the expensive attention pass then runs only over that subset. DeepSeek V4 and GLM 5.x use this to keep per-token cost nearly flat while the context grows to a million tokens. Quality holds up because most attention mass concentrates on few tokens anyway.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →