Norms & blocks

Residual (skip) connection

Adds a block’s input to its output, so each layer only learns a correction. The reason very deep nets train at all.

First page of Deep Residual Learning for Image RecognitionIntroduced inDec 2015Deep Residual Learning for Image RecognitionHe et al. · arXiv 1512.03385 ↗
The residual skip: blocks learn only a correctionattention+MLP+the skip carries the input around each block · this is why deep stacks train

The Add nodes threading through a transformer are residual connections: each attention or MLP block computes a delta that is added onto an untouched copy of its input. Gradients can flow straight through the addition, which is what makes 50-plus-layer models trainable. The chain of additions is often called the ‘residual stream’. The shared workspace every block reads from and writes to.

Adoption over time

Share of new models that have included residual connections over time.

93%2022202320242025202693%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

sentence-transformers/all-MiniLM-L6-v2 architecture graphsentence-transformers/all-MiniLM-L6-v2sentence-similarity · ↓ 253.3M · ♡ 6kOpen in visualizer cross-encoder/ms-marco-MiniLM-L6-v2 architecture graphcross-encoder/ms-marco-MiniLM-L6-v2text-ranking · ↓ 86.7M · ♡ 314Open in visualizer BAAI/bge-small-en-v1.5 architecture graphBAAI/bge-small-en-v1.5feature-extraction · ↓ 64.9M · ♡ 550Open in visualizer google/electra-base-discriminator architecture graphgoogle/electra-base-discriminator↓ 58.2M · ♡ 158Open in visualizer google-bert/bert-base-uncased architecture graphgoogle-bert/bert-base-uncasedfill-mask · ↓ 50.4M · ♡ 3kOpen in visualizer sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 architecture graphsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2sentence-similarity · ↓ 46.3M · ♡ 1kOpen in visualizer BAAI/bge-m3 architecture graphBAAI/bge-m3sentence-similarity · ↓ 38.0M · ♡ 3kOpen in visualizer sentence-transformers/all-mpnet-base-v2 architecture graphsentence-transformers/all-mpnet-base-v2sentence-similarity · ↓ 23.9M · ♡ 1kOpen in visualizer amazon/chronos-2 architecture graphamazon/chronos-2time-series-forecasting · ↓ 23.8M · ♡ 435Open in visualizer google-t5/t5-small architecture graphgoogle-t5/t5-smalltranslation · ↓ 23.7M · ♡ 600Open in visualizer Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 21.5M · ♡ 2kOpen in visualizer FacebookAI/xlm-roberta-base architecture graphFacebookAI/xlm-roberta-basefill-mask · ↓ 21.4M · ♡ 895Open in visualizer

View 4,267 catalog matches for Residual (skip) connection →

Related concepts

RMSNorm Attention residuals (AttnRes) Logit softcapping Gated MLP (SwiGLU) Hyper-connections (mHC) Transformer block

hfviewer renders the full architecture of 4,700+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →