Previous work has demonstrated the benefits of incorporating additional linguistic annotations such as syntactic trees into neural machine translation. However the cost of obtaining those syntactic annotations is expensive for many languages and the quality of unsupervised learning linguistic structures is too poor to be helpful. In this work, we aim to improve neural machine translation via source side dependency syntax but without explicit annotation. We propose a set of models that learn to induce dependency trees on the source side and learn to use that information on the target side. Importantly, we also show that our dependency trees capture important syntactic features of language and improve translation quality on two language pairs En-De and En-Ru.
This paper describes a method to induce source-side dependency structures in service to neural machine translation. The idea of learning soft dependency arcs in tandem with an NMT objective is very similar to recent notions of self-attention (Vaswani et al., 2017, cited) or previous work on latent graph parsing for NMT (Hashimoto and Tsuruoka, 2017, cited). This paper introduces three innovations: (1) they pass the self-attention scores through a matrix-tree theorem transformation to produce marginals over tree-constrained head probabilities; (2) they explicitly specify how the dependencies are to be used, meaning that rather than simply attending over dependency representations with a separate attention, they select a soft word to attend to through the traditional method, and then attend to that word’s soft head (called Shared Attention in the paper); and (3) they gate when attention is used. I feel that the first two ideas are particularly interesting. Unfortunately, the results of the NMT experiments are not particularly compelling, with overall gains over baseline NMT being between 0.6 and 0.8 BLEU. However, they include a useful ablation study that shows fairly clearly that both ideas (1) and (2) contribute equally to their modest gains, and that without them (FA-NMT Shared=No in Table 2), there would be almost no gains at all. Interesting side-experiments investigate their accuracy as a dependency parser, with and without a hard constraint on the system’s latent dependency decisions.
This paper has some very good ideas, and asks questions that are very much worth asking. In particular, the question of whether a tree constraint is useful in self-attention is very worthwhile. Unfortunately, this is mostly a negative result, with gains over “flat attention” being relatively small. I also like the “Shared Attention” - it makes a lot of sense to say that if the “semantic” attention mechanism has picked a particular word, one should also attend to that word’s head; it is not something I would have thought of on my own. The paper is also marred by somewhat weak writing, with a number of disfluencies and awkward phrasings making it somewhat difficult to follow.
In terms of specific criticisms:
I found the motivation section to be somewhat weak. We need a better reason than morphology to want to do source-side dependency parsing. All published error analyses of strong NMT systems (Bentivogli et al, EMNLP 2016; Toral and Sanchez-Cartagena, EACL 2017; Isabelle et al, EMNLP 2017) have shown that morphology is a strength, not a weakness of these systems, and the sorts of head selection problems shown in Figure 1 are, in my experience, handled capably by existing LSTM-based systems.
The paper mentions “significant improvements” in only two places: the introduction and the conclusion. With BLEU score differences being so low, the authors should specify how statistical significance is measured; ideally using a technique that accounts for the variance of random restarts (i.e.: Clark et al, ACL 2011).
Equation (3): I couldn’t find the definition for H anywhere.
Sentence before Equation (5): I believe there is a typo here, “f takes z_i” should be “f takes u_t”.
First section of Section 3: please cite the previous work you are talking about in this sentence.
My understanding was that the dependency marginals in p(z_{i,j}=1|x,\phi) in Equation (11) are directly used as \beta_{i,j}. If I’m correct, that’s probably worth spelling out explicitly in Equation (11): \beta_{i,j} = p(z_{i,j}=1|x,\phi) = …
I don’t don’t feel like the clause between equations (17) and (18), “when sharing attention weights from the decoder with the encoder” is a good description of your clever “shared attention” idea. In general, I found this region of the paper, including these two equations and the text between them, very difficult to follow.
Section 4.4: It’s very very good that you compared to “flat attention”, but it’s too bad for everyone cheering for linguistically-informed syntax that the results weren’t better.
Table 5: I had a hard time understanding Table 5 and the corresponding discussion. What are “production percentages”?
Finally, it would have been interesting to include the FA system in the dependency accuracy experiment (Table 4), to see if it made a big difference there.