Our qualitative analysis of the trained model
shows that it indeed captures the linguistic regularities in multiple levels i.e. at the word level as
well as phrase level. This suggests that there may
be more natural language related applications that
may beneﬁt from the proposed RNN Encoder–
Decoder.
The proposed architecture has large potential
for further improvement and analysis. One approach that was not investigated here is to replace the whole, or a part of the phrase table by
letting the RNN Encoder–Decoder propose target
phrases. Also, noting that the proposed model is
not limited to being used with written language,
it will be an important future research to apply the
proposed architecture to other applications such as
speech transcription.
Acknowledgments
KC, BM, CG, DB and YB would like to thank
NSERC, Calcul Qu ´ebec, Compute Canada, the
Canada Research Chairs and CIFAR. FB and HS
were partially funded by the European Commis-sion under the project MateCat, and by DARPA
under the BOLT project.
References
[Auli et al.2013] Michael Auli, Michel Galley, Chris
Quirk, and Geoffrey Zweig. 2013. Joint language
and translation modeling with recurrent neural networks. In Proceedings of the ACL Conference on
Empirical Methods in Natural Language Processing
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio