The paper proposes that the superior performance of Transformers in deep learning is due to an architectural bias towards mesa-optimization, a learned process within the forward pass. They reverse-engineer Transformers and show that the learned optimization algorithm can be used for few-shot tasks. They also propose a new self-attention layer that improves performance.