This August 2020 paper introduces linear transformers, a novel approach to addressing the computational and memory inefficiencies of traditional transformer models, particularly for long sequences. By reframing the self-attention mechanism using a linear dot-product of kernel feature maps, the authors reduce the computational complexity from quadratic to linear, enabling significantly faster autoregressive inference. The research highlights the relationship between transformers and recurrent neural networks (RNNs), demonstrating that a causally masked transformer can be expressed as an RNN, thus allowing for constant time and memory per prediction during inference. Experimental results across image generation and speech recognition tasks show that linear transformers achieve comparable performance to standard transformers while being orders of magnitude faster and requiring less memory.

Source: https://arxiv.org/pdf/2006.16236