tanh
Ws h1
;whereWs2Rnn.
The context vector ciare recomputed at each step by the alignment model:
ci=TxX
j=1ijhj;
13
Published as a conference paper at ICLR 2015
Model Updates (105)Epochs Hours GPU Train NLL Dev. NLL
RNNenc-30 8.46 6.4 109 TITAN BLACK 28.1 53.0
RNNenc-50 6.00 4.5 108 Quadro K-6000 44.0 43.6
RNNsearch-30 4.71 3.6 113 TITAN BLACK 26.7 47.2
RNNsearch-50 2.88 2.2 111 Quadro K-6000 40.7 38.1
RNNsearch-50?6.67 5.0 252 Quadro K-6000 36.7 35.2
Table 2: Learning statistics and relevant information. Each update corresponds to updating the
parameters once using a single minibatch. One epoch is one pass through the training set. NLL is
the average conditional log-probabilities of the sentences in either the training set or the development
set. Note that the lengths of the sentences differ.
where
ij=exp (eij)PTx
k=1exp (eik)
eij=v>
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio