with hyperparameters = 106and=
0:95(Zeiler, 2012). At each update, we used 64
randomly selected phrase pairs from a phrase table (which was created from 348M words). The
model was trained for approximately three days.
Details of the architecture used in the experiments are explained in more depth in the supplementary material.
4.1.2 Neural Language Model
In order to assess the effectiveness of scoring
phrase pairs with the proposed RNN Encoder–
Decoder, we also tried a more traditional approach
of using a neural network for learning a target
language model (CSLM) (Schwenk, 2007). Especially, the comparison between the SMT system
using CSLM and that using the proposed approach
of phrase scoring by RNN Encoder–Decoder will
clarify whether the contributions from multiple
neural networks in different parts of the SMT sys-tem add up or are redundant.
We trained the CSLM model on 7-grams
from the target corpus. Each input word
was projected into the embedding space R512,
and they were concatenated to form a 3072dimensional vector. The concatenated vector was
fed through two rectiﬁed layers (of size 1536 and
1024) (Glorot et al., 2011). The output layer was
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio