Norms & blocks

Attention residuals (AttnRes)

Blocks don’t just add onto one residual stream. They attend back over earlier depths and pull in a weighted mix of what is useful.

Goes back toFeb 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted AveragingPagliardini et al. · arXiv 2402.02622 ↗

A plain residual connection adds each block’s output onto a single running stream, so depth-to-depth information has to travel through that one shared channel. Attention residuals generalize the idea: a set of learned α operators lets a block look back across all of the earlier depths at once and retrieve a weighted combination of their representations. It is attention over depth, the way ordinary attention is over sequence. Moonshot’s Kimi K3 is the first open model to announce it at scale (the α operators in its diagram). The mechanism has roots in DenseFormer’s depth-weighted averaging of past layers, with the newer twist that the mixing weights are computed from the input rather than fixed after training.

Related concepts