translation in which most of input and output sentences are only 15–40 words. However, this may
limit the applicability of the proposed scheme to other tasks.
8
Published as a conference paper at ICLR 2015
6.2 N EURAL NETWORKS FOR MACHINE TRANSLATION
Since Bengio et al. (2003) introduced a neural probabilistic language model which uses a neural network to model the conditional probability of a word given a ﬁxed number of the preceding words,
neural networks have widely been used in machine translation. However, the role of neural networks has been largely limited to simply providing a single feature to an existing statistical machine
translation system or to re-rank a list of candidate translations provided by an existing system.
For instance, Schwenk (2012) proposed using a feedforward neural network to compute the score of
a pair of source and target phrases and to use the score as an additional feature in the phrase-based
statistical machine translation system. More recently, Kalchbrenner and Blunsom (2013) and Devlin
et al. (2014) reported the successful use of the neural networks as a sub-component of the existing
translation system. Traditionally, a neural network trained as a target-side language model has been
used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al. , 2006).
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio