The attention mechanism is an indispensable component of any state-of-the-art neural machine translation system. However, existing attention methods are often token-based and ignore the importance of phrasal alignments, which are the backbone of phrase-based statistical machine translation. We propose a novel phrasebased attention method to model n-grams of tokens as the basic attention entities, and design multi-headed phrasal attentions within the Transformer architecture to perform token-to-token and token-to-phrase mappings. Our approach yields improvements in English-German, English-Russian and English-French translation tasks on the standard WMT'14 test set. Furthermore, our phrasal attention method shows improvements on the one-billion-word language modeling benchmark.
This paper proposes an extension of the attention module that explicitly incorporates phrase information. Using convolution, attention scores are obtained independently for each n-gram type, and then combined. Transformer models with the proposed phrase attention are evaluated on multiple translation tasks, as well as on language modelling, generally obtaining better results than by simply increasing model size.
I lean towards the acceptance of the paper. The approach is fairly well motivated, likely easy to implement and results are mostly convincing. However, some claims may be too strong and I had difficulty understanding some parts of the approach.
I find the idea interesting. Standard attention is unbiased to distance (but sensitive to it because of positional embeddings). Phrasal attention may be a useful learning bias, giving particular importance to nearby words.
On 3 WMT'14 translation tasks, the proposed approach leads to improvements between 0.7 and 1.8 BLEU with respect to Transformer Base. Running each model with different random seeds and presenting statistical significance results would be ideal, but such runs can be expensive given the size of the datasets. Using phrasal attention appears to be more efficient than simply increasing model size. In addition to the number of parameters, the number of FLOPs per update might also be useful to know. Phrasal attention also leads to lower perplexity on a large-scale modeling task, although I can't confidently evaluate the importance of this result.
While results in the model interpretation section are cherry-picked, they illustrate that the model can use the additional capacity provided by phrasal attention. There is also clear qualitative differences between layers.
Some equations are confusing. For example, in Eq. 5, the right-most argument of Conv_n() appears to be of dimension nxdx1, but convolutions are defined for dimension nxdxd. I would suggest going over the presentation of phrasal attention carefully (or correct me if I interpreted the notation wrongly).
Some claims made in the paper may be too strong. While there are similarities between alignment and attention, they are not necessarily interchangeable in neural models. For example, (Koehn and Knowles. Six Challenges for Neural Machine Translation) show that they can be mismatched (Fig. 9).
Moreover, while input embeddings (and arguably the last decoder hidden layer) mostly contain token-level information, intermediate representations merge information from multiple positions. As such, at a given layer, it is not guaranteed the the i^{th} vector is a representation i^{th} token. For example, (Voita et al. The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives) show that the mutual information between an input token and its corresponding encoder representation diminishes as depth increases. As such, neighbouring representations may not represent n-grams.
It would be appropriate to compare the proposed approach to (Hao et al. Multi-Granularity Self-Attention for Neural Machine Translation). However, this is very recent work (September 5 on ArXiv), so it would be understandable for the authors not to know about it.
Questions:
While searching for related work, I found an earlier submitted version of this paper ("Phrase-Based Attentions", submitted to ICLR 2019). The reported numbers differ from the current version. Why?