Attention

Multi-head latent attention (MLA)

Compresses keys and values into one small latent vector per token. The cache stores the latent, not full K/V.

First page of DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelIntroduced inMay 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelDeepSeek-AI · arXiv 2405.04434 ↗
Multi-head latent attentionfull K/V per tokenKVcompress(low-rank)cache stores tiny latentsexpand on use

Introduced by DeepSeek, MLA projects each token’s keys and values down into a compact latent vector and reconstructs per-head K/V from it on the fly. Only the small latent is kept in the KV cache, cutting cache memory by an order of magnitude versus storing full keys and values, which is what makes very long contexts affordable. A small decoupled RoPE branch keeps position information that the compression would otherwise destroy.

Adoption over time

Share of new models that have included MLA over time.

19%2022202320242025202619%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 8.7M · ♡ 13kOpen in visualizer zai-org/GLM-5.2-FP8 architecture graphzai-org/GLM-5.2-FP8text-generation · ↓ 3.0M · ♡ 223Open in visualizer deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 3.0M · ♡ 2kOpen in visualizer zai-org/GLM-4.7-Flash architecture graphzai-org/GLM-4.7-Flashtext-generation · ↓ 2.4M · ♡ 2kOpen in visualizer zai-org/GLM-5-FP8 architecture graphzai-org/GLM-5-FP8text-generation · ↓ 1.5M · ♡ 181Open in visualizer deepseek-ai/DeepSeek-V4-Pro architecture graphdeepseek-ai/DeepSeek-V4-Protext-generation · ↓ 1.5M · ♡ 5kOpen in visualizer nvidia/GLM-5.2-NVFP4 architecture graphnvidia/GLM-5.2-NVFP4text-generation · ↓ 1.3M · ♡ 273Open in visualizer moonshotai/Kimi-K2.6 architecture graphmoonshotai/Kimi-K2.6image-text-to-text · ↓ 1.3M · ♡ 2kOpen in visualizer deepseek-ai/DeepSeek-V2-Lite-Chat architecture graphdeepseek-ai/DeepSeek-V2-Lite-Chattext-generation · ↓ 1.2M · ♡ 144Open in visualizer deepseek-ai/DeepSeek-V3.2 architecture graphdeepseek-ai/DeepSeek-V3.2text-generation · ↓ 1.2M · ♡ 1kOpen in visualizer nvidia/DeepSeek-V4-Flash-NVFP4 architecture graphnvidia/DeepSeek-V4-Flash-NVFP4text-generation · ↓ 1.1M · ♡ 71Open in visualizer moonshotai/Kimi-K2.5 architecture graphmoonshotai/Kimi-K2.5image-text-to-text · ↓ 1.1M · ♡ 3kOpen in visualizer

Browse all 92 models with this in the catalog →

Related concepts

QK-Norm Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →