Attention

Multi-head latent attention (MLA)

Compresses keys and values into one small latent vector per token. The cache stores the latent, not full K/V.

First page of DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelIntroduced inMay 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelDeepSeek-AI · arXiv 2405.04434 ↗
Multi-head latent attentionfull K/V per tokenKVcompress(low-rank)cache stores tiny latentsexpand on use

Introduced by DeepSeek, MLA projects each token’s keys and values down into a compact latent vector and reconstructs per-head K/V from it on the fly. Only the small latent is kept in the KV cache, cutting cache memory by an order of magnitude versus storing full keys and values, which is what makes very long contexts affordable. A small decoupled RoPE branch keeps position information that the compression would otherwise destroy.

Adoption over time

Share of new models that have included MLA over time.

9%202220232024202520269%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 4.4M · ♡ 4kOpen in visualizer moonshotai/Kimi-K3 architecture graphmoonshotai/Kimi-K3image-text-to-text · ↓ 2.4M · ♡ 11kOpen in visualizer zai-org/GLM-4.7-Flash architecture graphzai-org/GLM-4.7-Flashtext-generation · ↓ 1.9M · ♡ 2kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 1.8M · ♡ 2kOpen in visualizer deepseek-ai/DeepSeek-V3.2 architecture graphdeepseek-ai/DeepSeek-V3.2text-generation · ↓ 1.6M · ♡ 1kOpen in visualizer zai-org/GLM-5.2-FP8 architecture graphzai-org/GLM-5.2-FP8text-generation · ↓ 1.3M · ♡ 265Open in visualizer deepseek-ai/DeepSeek-V3 architecture graphdeepseek-ai/DeepSeek-V3text-generation · ↓ 1.0M · ♡ 4kOpen in visualizer deepseek-ai/DeepSeek-V3-0324 architecture graphdeepseek-ai/DeepSeek-V3-0324text-generation · ↓ 986k · ♡ 3kOpen in visualizer zai-org/GLM-5.2 architecture graphzai-org/GLM-5.2text-generation · ↓ 965k · ♡ 5kOpen in visualizer nvidia/GLM-5.2-NVFP4 architecture graphnvidia/GLM-5.2-NVFP4text-generation · ↓ 940k · ♡ 321Open in visualizer deepseek-ai/DeepSeek-R1 architecture graphdeepseek-ai/DeepSeek-R1text-generation · ↓ 777k · ♡ 14kOpen in visualizer deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct architecture graphdeepseek-ai/DeepSeek-Coder-V2-Lite-Instructtext-generation · ↓ 744k · ♡ 653Open in visualizer

View 194 catalog matches for Multi-head latent attention (MLA) →

Related concepts

QK-Norm Sparse attention (top-k token selection) Sliding-window attention Linear attention (gated DeltaNet) Mamba / state-space block (SSM) Short convolution block Grouped-query attention (GQA) Self-attention

hfviewer renders the full architecture of 4,700+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →