Norms & blocks

Attention residuals (AttnRes)

Blocks don’t just add onto one residual stream. They attend back over earlier depths and pull in a weighted mix of what is useful.

Goes back toFeb 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted AveragingPagliardini et al. · arXiv 2402.02622 ↗

A plain residual connection adds each block’s output onto a single running stream, so depth-to-depth information has to travel through that one shared channel. Attention residuals generalize the idea: a set of learned α operators lets a block look back across all of the earlier depths at once and retrieve a weighted combination of their representations. It is attention over depth, the way ordinary attention is over sequence. Moonshot’s Kimi K3 is the first open model to announce it at scale (the α operators in its diagram). The mechanism has roots in DenseFormer’s depth-weighted averaging of past layers, with the newer twist that the mixing weights are computed from the input rather than fixed after training.

First traced in Kimi K3 (released July 27): the AttnRes depth-mixing edges are visible in its live traced graph — the first open model we have traced that uses this.

Adoption over time

Share of new models that have included this block over time.

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Related concepts