Norms & blocks

Hyper-connections (mHC)

Widens the single residual stream into several parallel streams with learned mixing between them.

First page of Hyper-ConnectionsIntroduced inSep 2024Hyper-ConnectionsZhu et al. · arXiv 2409.19606 ↗
Hyper-connections: a wider residual streamone stream becomes several in parallel, with learned mixing between themone streamn streams, learned mixing

Hyper-connections replace the transformer’s single residual stream with several parallel streams; at each block, learned matrices mix the streams before and after the block runs. DeepSeek V4’s mHC variant constrains that mixing to be doubly stochastic (computed with a few Sinkhorn iterations) so the streams stay balanced. The effect is richer gradient routing than one shared residual. One of the most distinctive pieces of the V4 architecture.

Adoption over time

Share of new models that have included hyper-connections over time.

1%202220232024202520261%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

zai-org/GLM-5.3-Flash architecture graphzai-org/GLM-5.3-Flashimage-text-to-text · ↓ 5.4M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-0731 architecture graphdeepseek-ai/DeepSeek-V4-Flash-0731text-generation · ↓ 4.5M · ♡ 4kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash architecture graphdeepseek-ai/DeepSeek-V4-Flashtext-generation · ↓ 1.6M · ♡ 2kOpen in visualizer deepseek-ai/DeepSeek-V4.1-Flash architecture graphdeepseek-ai/DeepSeek-V4.1-Flashimage-text-to-text · ↓ 1.3M · ♡ 3kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-DSpark architecture graphdeepseek-ai/DeepSeek-V4-Flash-DSparktext-generation · ↓ 1.0M · ♡ 281Open in visualizer deepseek-ai/DeepSeek-V4-Flash-Vision-Exp architecture graphdeepseek-ai/DeepSeek-V4-Flash-Vision-Expimage-text-to-text · ↓ 770k · ♡ 925Open in visualizer deepseek-ai/DeepSeek-V4-Pro architecture graphdeepseek-ai/DeepSeek-V4-Protext-generation · ↓ 569k · ♡ 6kOpen in visualizer deepseek-ai/DeepSeek-V4-Flash-Base architecture graphdeepseek-ai/DeepSeek-V4-Flash-Base↓ 556k · ♡ 322Open in visualizer sgl-project/DeepSeek-V4-Flash-FP8 architecture graphsgl-project/DeepSeek-V4-Flash-FP8↓ 348k · ♡ 15Open in visualizer deepseek-ai/DeepSeek-V4-Pro-0813 architecture graphdeepseek-ai/DeepSeek-V4-Pro-0813text-generation · ↓ 160k · ♡ 842Open in visualizer 0xSero/deepseek-v4-flash-0731-spark architecture graph0xSero/deepseek-v4-flash-0731-sparktext-generation · ↓ 144k · ♡ 50Open in visualizer nvidia/DeepSeek-V4-Pro-NVFP4 architecture graphnvidia/DeepSeek-V4-Pro-NVFP4text-generation · ↓ 138k · ♡ 77Open in visualizer

View 98 catalog matches for Hyper-connections (mHC) →

Related concepts

RMSNorm Attention residuals (AttnRes) Residual (skip) connection Logit softcapping Gated MLP (SwiGLU) Transformer block

hfviewer renders the full architecture of 6,700+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →