the activation functions fof recurrent neural networks (RNN) and the alignment model a. Here, we
describe the choices we made for the experiments in this paper.
A.1.1 R ECURRENT NEURAL NETWORK
For the activation function fof an RNN, we use the gated hidden unit recently proposed by Cho
et al. (2014a). The gated hidden unit is an alternative to the conventional simple units such as an
element-wise tanh . This gated unit is similar to a long short-term memory (LSTM) unit proposed
earlier by Hochreiter and Schmidhuber (1997), sharing with it the ability to better model and learn
long-term dependencies. This is made possible by having computation paths in the unfolded RNN
for which the product of derivatives is close to 1. These paths allow gradients to ﬂow backward
easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al. , 1994;
Pascanu et al. , 2013a). It is therefore possible to use LSTM units instead of the gated hidden unit
described here, as was done in a similar context by Sutskever et al. (2014).
The new state siof the RNN employing ngated hidden units8is computed by
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio