Norms & blocks

Residual (skip) connection

Adds a block’s input to its output, so each layer only learns a correction. The reason very deep nets train at all.

First page of Deep Residual Learning for Image RecognitionIntroduced inDec 2015Deep Residual Learning for Image RecognitionHe et al. · arXiv 1512.03385 ↗
The residual skip: blocks learn only a correctionattention+MLP+the skip carries the input around each block · this is why deep stacks train

The Add nodes threading through a transformer are residual connections: each attention or MLP block computes a delta that is added onto an untouched copy of its input. Gradients can flow straight through the addition, which is what makes 50-plus-layer models trainable. The chain of additions is often called the ‘residual stream’. The shared workspace every block reads from and writes to.

Adoption over time

Share of new models that have included residual connections over time.

100%20222023202420252026100%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

sentence-transformers/all-MiniLM-L6-v2 architecture graphsentence-transformers/all-MiniLM-L6-v2sentence-similarity · ↓ 248.5M · ♡ 5kOpen in visualizer cross-encoder/ms-marco-MiniLM-L6-v2 architecture graphcross-encoder/ms-marco-MiniLM-L6-v2text-ranking · ↓ 83.9M · ♡ 285Open in visualizer google-bert/bert-base-uncased architecture graphgoogle-bert/bert-base-uncasedfill-mask · ↓ 78.4M · ♡ 3kOpen in visualizer BAAI/bge-small-en-v1.5 architecture graphBAAI/bge-small-en-v1.5feature-extraction · ↓ 65.2M · ♡ 514Open in visualizer google/electra-base-discriminator architecture graphgoogle/electra-base-discriminator↓ 54.6M · ♡ 142Open in visualizer sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 architecture graphsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2sentence-similarity · ↓ 48.9M · ♡ 1kOpen in visualizer BAAI/bge-m3 architecture graphBAAI/bge-m3sentence-similarity · ↓ 34.7M · ♡ 3kOpen in visualizer sentence-transformers/all-mpnet-base-v2 architecture graphsentence-transformers/all-mpnet-base-v2sentence-similarity · ↓ 29.7M · ♡ 1kOpen in visualizer google-t5/t5-small architecture graphgoogle-t5/t5-smalltranslation · ↓ 27.2M · ♡ 576Open in visualizer Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 26.1M · ♡ 1kOpen in visualizer FacebookAI/xlm-roberta-base architecture graphFacebookAI/xlm-roberta-basefill-mask · ↓ 23.5M · ♡ 873Open in visualizer openai/clip-vit-base-patch32 architecture graphopenai/clip-vit-base-patch32zero-shot-image-classification · ↓ 23.4M · ♡ 982Open in visualizer

Browse all 2166 models with this in the catalog →

Related concepts

RMSNorm Attention residuals (AttnRes) Logit softcapping Gated MLP (SwiGLU) Hyper-connections (mHC) Transformer block

hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →