Attention Is All You Need

🤖 Plain-English Summary

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. On the WMT 2014 English-to-French translation task, our model establishes a new single-model advanced BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature.

🔑 Key Findings

• The best performing models also connect the encoder and decoder through an attention mechanism.
• We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
• Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.

🔗 Related Resources