twice: ﬁrst with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then
with the sentences of length up to 50 word (RNNencdec-50, RNNsearch-50).
The encoder and decoder of the RNNencdec have 1000 hidden units each.7The encoder of the
RNNsearch consists of forward and backward recurrent neural networks (RNN) each having 1000
hidden units. Its decoder has 1000 hidden units. In both cases, we use a multilayer network with a
single maxout (Goodfellow et al. , 2013) hidden layer to compute the conditional probability of each
target word (Pascanu et al. , 2014).
We use a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler,
2012) to train each model. Each SGD update direction is computed using a minibatch of 80 sentences. We trained each model for approximately 5 days.
Once a model is trained, we use a beam search to ﬁnd a translation that approximately maximizes the
conditional probability (see, e.g., Graves, 2012; Boulanger-Lewandowski et al. , 2013). Sutskever
et al. (2014) used this approach to generate translations from their neural machine translation model.
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio