CIFAR Senior Fellow
1arXiv:1409.0473v7  [cs.CL]  19 May 2016
Published as a conference paper at ICLR 2015
The most important distinguishing feature of this approach from the basic encoder–decoder is that
it does not attempt to encode a whole input sentence into a single ﬁxed-length vector. Instead, it encodes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively
while decoding the translation. This frees a neural translation model from having to squash all the
information of a source sentence, regardless of its length, into a ﬁxed-length vector. We show this
allows a model to cope better with long sentences.
In this paper, we show that the proposed approach of jointly learning to align and translate achieves
signiﬁcantly improved translation performance over the basic encoder–decoder approach. The improvement is more apparent with longer sentences, but can be observed with sentences of any
length. On the task of English-to-French translation, the proposed approach achieves, with a single
model, a translation performance comparable, or close, to the conventional phrase-based system.
Furthermore, qualitative analysis reveals that the proposed model ﬁnds a linguistically plausible
(soft-)alignment between a source sentence and the corresponding target sentence.
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio