Generation reuses cached keys/values of all previous tokens instead of recomputing them - memory becomes the bottleneck.
When a model generates token by token, the keys and values of every earlier token are stored so each new token only computes attention against the cache instead of re-running the whole sequence. The cache grows with context length × layers × KV heads × head dim, and it - not compute - is usually what limits batch size and context in serving. Most modern attention variants (GQA, MLA, sliding windows, linear attention) are primarily KV-cache-size optimizations.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →