distribution of mean 0and variance 0:0012. All the elements of Vaand all the bias vectors were initialized to zero. Any other weight matrix was initialized by sampling from the Gaussian distribution
of mean 0and variance 0:012.
B.2 T RAINING
We used the stochastic gradient descent (SGD) algorithm. Adadelta (Zeiler, 2012) was used to
automatically adapt the learning rate of each parameter ( = 106and= 0:95). We explicitly
14
Published as a conference paper at ICLR 2015
normalized the L2-norm of the gradient of the cost function each time to be at most a predeﬁned
threshold of 1, when the norm was larger than the threshold (Pascanu et al. , 2013b). Each SGD
update direction was computed with a minibatch of 80 sentences.
At each update our implementation requires time proportional to the length of the longest sentence in
a minibatch. Hence, to minimize the waste of computation, before every 20-th update, we retrieved
1600 sentence pairs, sorted them according to the lengths and split them into 20 minibatches. The
training data was shufﬂed once before training and was traversed sequentially in this manner.
In Tables 2 we present the statistics related to training all the models used in the experiments.
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio