Compresses keys and values into one small latent vector per token - the cache stores the latent, not full K/V.
Introduced by DeepSeek, MLA projects each token's keys and values down into a compact latent vector and reconstructs per-head K/V from it on the fly. Only the small latent is kept in the KV cache, cutting cache memory by an order of magnitude versus storing full keys and values, which is what makes very long contexts affordable. A small decoupled RoPE branch keeps position information that the compression would otherwise destroy.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →