The paper introduces Dynamic Tanh (DyT), a novel element-wise operation designed to replace normalization layers in Transformer models. Traditionally, normalization layers like Layer Normalization (LN) are considered essential for stable and effective training of deep neural networks. However, this paper challenges that belief by demonstrating that DyT, which mimics the S-shaped input-output mapping observed in LN, can achieve comparable or superior performance across various tasks, including vision, language, and speech processing, often without extensive hyperparameter tuning. This research suggests that the non-linear squashing of extreme values by normalization layers, rather than their statistical normalization, is a key mechanism, offering new insights into their role in deep learning architectures. While effective for Transformers, preliminary findings suggest DyT may not directly substitute Batch Normalization (BN) in classic ConvNets.

Source: Published June 2025

https://arxiv.org/pdf/2503.10622