and~ti;kis thek-th element of a vector ~tiwhich is computed by
~ti=Uosi1+VoEyi1+Coci:
Wo2RKyl,Uo2R2ln,Vo2R2lmandCo2R2l2nare weight matrices. This can be understood as having a deep output (Pascanu et al. , 2014) with a single maxout hidden layer (Goodfellow
et al. , 2013).
A.2.3 M ODEL SIZE
For all the models used in this paper, the size of a hidden layer nis 1000, the word embedding
dimensionality mis 620 and the size of the maxout hidden layer in the deep output lis 500. The
number of hidden units in the alignment model n0is 1000.
B T RAINING PROCEDURE
B.1 P ARAMETER INITIALIZATION
We initialized the recurrent weight matrices U;Uz;Ur; U; Uz; Ur; !U; !Uzand !Uras random orthogonal matrices. For WaandUa, we initialized them by sampling each element from the Gaussian
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio