Architecture timeline

How we got to DeepSeek‑V4‑Pro.

A decade of architectural ideas, each one small on its own, adding up to a frontier model. Every stop links to a plain-English explanation or a live interactive graph.

17 stops 12 papers 7 model graphs 2015 to 2026
Concept Model Paper and model in one release
Foundations2015 to 2017

Three ideas arrive before anyone knows what they will become.

ConceptDec 2015Residual connectionsDeep networks only train when each layer learns a small correction on top of its input. Every model on this page is a stack of blocks wired through this one trick.attn+MLP+ Deep Residual Learning for Image RecognitionarXiv 1512.03385 · Dec 2015Read the glossary entry →
ConceptJan 2017Mixture of expertsWhy run the whole network for every token? A router wakes only a few expert MLPs at a time. The idea arrives years early and waits for its moment.routershared Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerarXiv 1701.06538 · Jan 2017MoE routerRead the glossary entry →
ConceptJun 2017Self-attentionTokens build queries, keys and values and gather context directly from each other. The transformer block becomes the repeating unit of modern AI. Attention Is All You NeedarXiv 1706.03762 · Jun 2017Transformer blockRead the glossary entry →
The recipe assembles2019 to 2021

Scaling works, and the modern decoder block picks up its parts one paper at a time.

ModelFeb 2019openai-community/gpt2Scale a plain decoder stack and let it write. Generation reuses cached keys and values, so the KV cache starts its career as the memory bottleneck of inference.openai-community/gpt2 architecture graphtext-generation · ↓ 14.0M · ♡ 3kKV cacheOpen in visualizer →
ConceptOct 2019RMSNormLayerNorm without the mean-centering: rescale by magnitude only. Cheaper, simpler, and soon the default normalizer of open models.× gainRMSRMS = 1 Root Mean Square Layer NormalizationarXiv 1910.07467 · Oct 2019Read the glossary entry →
ConceptFeb 2020Gated MLP (SwiGLU)The feed-forward network splits into a gate and an up projection multiplied together. That is why you see three Linears and a Mul in every modern graph.gate_proj · siluup_proj× GLU Variants Improve TransformerarXiv 2002.05202 · Feb 2020Read the glossary entry →
ConceptApr 2021Rotary position embedding (RoPE)Positions become rotations of query and key vectors, and relative distance falls out of the math for free. The last piece of the standard recipe.same anglesame gap, same angleonly distance matters RoFormer: Enhanced Transformer with Rotary Position EmbeddingarXiv 2104.09864 · Apr 2021Read the glossary entry →
Open weights, efficient inference2023

The recipe goes open source, then learns to run cheap and long.

ModelFeb 2023huggyllama/llama-7bRMSNorm, SwiGLU and RoPE ship together in open weights. This exact recipe becomes the ancestor of nearly every open model since.huggyllama/llama-7b architecture graphtext-generation · ↓ 269k · ♡ 359Open in visualizer →
ConceptMay 2023Grouped-query attention (GQA)Several query heads share one key/value head. The KV cache shrinks several-fold and long contexts stop being a luxury.MHAGQA GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsarXiv 2305.13245 · May 2023Read the glossary entry →
ConceptAug 2023RoPE scaling (YaRN)Rescale RoPE frequencies and a model trained on short contexts operates on much longer ones. Context windows jump from thousands of tokens to hundreds of thousands.4k128k YaRN: Efficient Context Window Extension of Large Language ModelsarXiv 2309.00071 · Aug 2023Read the glossary entry →
ModelSep 2023mistralai/Mistral-7B-v0.1GQA plus sliding-window attention in a 7B that punched far above its size. Efficient attention becomes table stakes.mistralai/Mistral-7B-v0.1 architecture graphtext-generation · ↓ 858k · ♡ 4kSliding-window attentionOpen in visualizer →
ModelDec 2023mistralai/Mixtral-8x7B-v0.1The 2017 idea lands in an open flagship: eight experts per layer, two active per token. Capacity grows without the compute bill.mistralai/Mixtral-8x7B-v0.1 architecture graph↓ 50k · ♡ 2kMoE expertsOpen in visualizer →
The DeepSeek stack2024 to 2025

One lab ships its research as models, and the cache, the experts and the attention map all get rebuilt.

ConceptJan 2024Shared expertDeepSeekMoE slices experts fine and adds one that never turns off: every token passes through the shared expert, and the router only tops it up.routershared DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsarXiv 2401.06066 · Jan 2024Read the glossary entry →
Paper + modelMay 2024deepseek-ai/DeepSeek-V2The paper and the model are the same release. Keys and values compress into one small latent per token, and the cache shrinks by an order of magnitude.deepseek-ai/DeepSeek-V2 architecture graphtext-generation · ↓ 7k · ♡ 334DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelarXiv 2405.04434 · May 2024Multi-head latent attentionOpen in visualizer →
Paper + modelDec 2024deepseek-ai/DeepSeek-V3Multi-token prediction trains richer representations while grouped expert routing keeps 256 experts friendly to real hardware. The open frontier model of its winter.deepseek-ai/DeepSeek-V3 architecture graphtext-generation · ↓ 1.0M · ♡ 4kDeepSeek-V3 Technical ReportarXiv 2412.19437 · Dec 2024Multi-token predictionGrouped expert routingOpen in visualizer →
ConceptSep 2025Sparse attention returnsA 2019 idea, back for the long-context era: a cheap indexer scores past tokens and full attention runs only over the winners. Generating Long Sequences with Sparse TransformersarXiv 1904.10509 · Apr 2019Read the glossary entry →
2026 · where it all lands

deepseek-ai/DeepSeek-V4-Pro

Every idea above, alive in one graph. Open it and find each block yourself: hover any node and hfviewer explains it with this model’s real numbers.

deepseek-ai/DeepSeek-V4-Pro architecture graph
Residual connectionsRMSNormSwiGLURoPEMoE expertsShared expertMLASparse attentionMulti-token predictionSliding window
Open DeepSeek-V4-Pro in the visualizer →

Want the definitions behind each stop? The glossary explains all 39 concepts, and the catalog has interactive graphs for 2,300+ models.

AlsoHow we got to FLUX.2 · the image generation timeline →AlsoHow we got to Qwen3-VL · the vision-language timeline →