Widens the single residual stream into several parallel streams with learned mixing between them.
Hyper-connections replace the transformer's single residual stream with several parallel streams; at each block, learned matrices mix the streams before and after the block runs. DeepSeek V4's mHC variant constrains that mixing to be doubly stochastic (computed with a few Sinkhorn iterations) so the streams stay balanced. The effect is richer gradient routing than one shared residual - one of the most distinctive pieces of the V4 architecture.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →