6 R ELATED WORK
6.1 L EARNING TO ALIGN
A similar approach of aligning an output symbol with an input symbol was proposed recently by
Graves (2013) in the context of handwriting synthesis. Handwriting synthesis is a task where the
model is asked to generate handwriting of a given sequence of characters. In his work, he used a
mixture of Gaussian kernels to compute the weights of the annotations, where the location, width
and mixture coefﬁcient of each kernel was predicted from an alignment model. More speciﬁcally,
his alignment was restricted to predict the location such that the location increases monotonically.
The main difference from our approach is that, in (Graves, 2013), the modes of the weights of the
annotations only move in one direction. In the context of machine translation, this is a severe limitation, as (long-distance) reordering is often needed to generate a grammatically correct translation
(for instance, English-to-German).
Our approach, on the other hand, requires computing the annotation weight of every word in the
source sentence for each word in the translation. This drawback is not severe with the task of
translation in which most of input and output sentences are only 15–40 words. However, this may
limit the applicability of the proposed scheme to other tasks.
8
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio