Adds a block's input to its output, so each layer only learns a correction - the reason very deep nets train at all.
The Add nodes threading through a transformer are residual connections: each attention or MLP block computes a delta that is added onto an untouched copy of its input. Gradients can flow straight through the addition, which is what makes 50-plus-layer models trainable. The chain of additions is often called the 'residual stream' - the shared workspace every block reads from and writes to.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →