The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
主流的序列转换模型基于复杂的循环神经网络或卷积神经网络，这些网络包含一个编码器和一个解码器。表现最佳的模型还通过注意力机制将编码器和解码器连接起来。我们提出了一种全新的简单网络架构——Transformer，它完全基于注意力机制，彻底摒弃了循环和卷积。在两个机器翻译任务上的实验表明，这些模型在质量上更胜一筹，同时具有更高的并行性，训练所需时间也显著减少。在 WMT 2014 年英德翻译任务中，我们的模型达到了 28.4 的 BLEU 值，比现有最佳结果（包括集成模型）高出 2 个多 BLEU 值。在 WMT 2014 年英法翻译任务中，我们的模型在 8 块 GPU 上训练 3.5 天后，创造了新的单模型 BLEU 得分纪录，达到 41.8，这仅为文献中最佳模型训练成本的一小部分。我们通过将 Transformer 成功应用于具有大量和有限训练数据的英语成分句法分析任务，证明了它在其他任务上也具有良好的通用性。
前往小宇宙评论区与主播互动
