depend oni, we can pre-compute it in advance to minimize the computational cost.
8Here, we show the formula of the decoder. The same formula can be used in the encoder by simply
ignoring the context vector ciand the related terms.
12
Published as a conference paper at ICLR 2015
A.2 D ETAILED DESCRIPTION OF THE MODEL
A.2.1 E NCODER
In this section, we describe in detail the architecture of the proposed model (RNNsearch) used in the
experiments (see Sec. 4–5). From here on, we omit all bias terms in order to increase readability.
The model takes a source sentence of 1-of-K coded word vectors as input
x= (x1;:::;x Tx); xi2RKx
and outputs a translated sentence of 1-of-K coded word vectors
y= (y1;:::;y Ty); yi2RKy;
whereKxandKyare the vocabulary sizes of source and target languages, respectively. TxandTy
respectively denote the lengths of source and target sentences.
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:
 !hi=(
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio