Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns
to align and translate simultaneously.
In the Encoder–Decoder framework, an encoder reads the input sentence, a sequence of vectors
x= (x1;;xTx), into a vector c.2The most common approach is to use an RNN such that
ht=f(xt;ht1) (1)
and
c=q(fh1;;hTxg);
whereht2Rnis a hidden state at time t, andcis a vector generated from the sequence of the
hidden states. fandqare some nonlinear functions. Sutskever et al. (2014) used an LSTM as fand
q(fh1;;hTg) =hT, for instance.
1We mean by the state-of-the-art performance, the performance of the conventional phrase-based system
without using any neural network-based component.
2Although most of the previous works (see, e.g., Cho et al. , 2014a; Sutskever et al. , 2014; Kalchbrenner and
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio