This episode examines "Attention Residuals," a March 2026 paper from Moonshot AI's Kimi team that challenges a foundational element of transformer architecture. The paper proposes replacing the fixed, uniform residual connections inherited from ResNet with learned attention mechanisms for depth-wise information aggregation. While attention replaced recurrent neural networks for sequence modeling over a decade ago, the authors argue that depth-wise aggregation remains stuck with the same fixed summation from 2015, creating an architectural asymmetry where learned selection should be used instead. The episode traces the dual role of residual connections — serving both as gradient highways for backpropagation and as information aggregation mechanisms — and explains why the latter has become a performance bottleneck in deep transformers.

The discussion centers on the PreNorm dilution problem, where unnormalized residual accumulation causes hidden-state magnitudes to grow linearly with network depth, progressively burying individual layer contributions under an ever-growing pile of summed vectors. In PreNorm architectures, despite their superior gradient stability during training, deep layer outputs contribute only one percent of the total magnitude at layer 100. This dilution effect helps explain why layer pruning experiments often show minimal performance loss when removing significant fractions of trained layers. The Kimi team's solution replaces fixed unit-weight summation with softmax attention over previous layer outputs, where each layer learns a single query vector to compute content-dependent weights that sum to one, maintaining bounded magnitude while enabling selective information aggregation.

The episode examines experimental validation across multiple model scales, from 460 million to 7 billion parameters, trained on datasets up to 100 billion tokens. Results show consistent improvements in perplexity and downstream task performance, with particularly strong gains in deeper models where dilution effects are most severe. The architecture introduces minimal computational overhead — approximately 3 percent — by sharing key-value projections with the self-attention sublayer and caching attention weights across the depth dimension. Design variants including bidirectional attention and explicit current-layer queries are explored, with causal attention and implicit queries recommended as the optimal configuration for both performance and efficiency in production deployments.

Interactive Visualization: Rethinking Residual Connections: Learned Attention Across Transformer Depth