Most state-of-the-art neural machine translation systems, despite being different in architectural skeletons (e.g., recurrence, convolutional), share an indispensable feature: the Attention. However, most existing attention methods are token-based and ignore the importance of phrasal alignments, the key ingredient for the success of phrase-based statistical machine translation. In this paper, we propose novel phrase-based attention methods to model n-grams of tokens as attention entities. We incorporate our phrase-based attentions into the recently proposed Transformer network, and demonstrate that our approach yields improvements of up to 1.3 and 0.5 BLEU points in English-to-German and German-to-English translation tasks, and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-toEnglish translation tasks on WMT'14 test set using WMT'16 training data.
The authors propose to include phrases (contiguous n-grams of wordpieces) in both the self-attention and encoder-decoder attention modules of the Transformer model (Vaswani et al., 2017). In standard multi-head attention, the logits of the attention distribution of each head is computed as the dot-product between query and key representations, which are position-specific. In the phrase-based attention proposed here, a convolution is first computed over the query, key, value sequences before logits are computed as before (a few variants of this scheme are explored). Results show an improvement of up to 1.3 BLEU points compared to the baseline Transformer model. However, the lack of a controlled experiment sheds substantial doubt on the efficiacy of the model (see below).
Contributions
-------------------
Proposes a simple way to incorporate n-grams in the Transformer model. The implementation is straightforward and should be fully replicable in an afternoon.
Having an inductive bias towards modeling of longer phrases seems intuitively useful, in particular when using subword representations, where subword units are often ambiguous. This is also motivated by the fact that prior work has shown that subword regularization, where sampling different subword segmentations during training can be useful.
Improvements in BLEU scores are quite strong.
Issues
---------
The experiments do not control for parameter count. The phrasal attention model adds significant number of parameters (e.g., "interleaved attention" corresponds to 3x the number of 1D convolution parameters in the attention layer). It is well established that more parameters correspond to increased BLEU scores (e.g., the 2x parameter count in the "big" Transformer setting from Vaswani et al. (2017) results in over 1 BLEU point improvement). This needs to be fixed!
The model is a very modest extension of the original Transformer model and so its value to the community beyond improved numbers is somewhat questionable.
While an explicit inductive bias for phrases seem plausible, it may be that this can already be fully captured by multi-head attention. With positional encodings, two heads can easily attend to adjacent positions which gives the model the same capacity as the convolutional phrase model. The result in the paper that trigrams do not add anything on top of bigrams signals to me that the model is already implicitly capturing phrase-level aspects in the multi-head attention. I would urge the authors to verify this by looking at gradient information (https://arxiv.org/abs/1312.6034).
There are several unsubstantiated claims: "Without specific attention to phrases, a particular attention function has to depend entirely on the token-level softmax scores of a phrase for phrasal alignment, which is not robust and reliable, thus making it more difficult to learn the mappings." - The attention is positional, but not necessarily token-based. The model has capacity to represent phrases in subsequent layers. WIth h heads , a position in the k-th layer can in principle represent h^k grams (each slot in layer 2 can represent a h-gram and so on).
The differences in training setup compared to Vaswani et al. (2017) needs to be explicit ("most of the training settings" is too handwavy). Please list any differences.
The notation is somewhat cumbersome and could use some polishing. For example, the input and output symbols both range over indices in [1,n]. The multi-head attention formulas also do not match the ones from Vaswani et al. (2017) fully. Please ensure consistency and readability of symbols and formulas.
The model inspection would be much improved by variance analysis. For example, the numbers in table 3 would be more useful if accompanied by variance across training runs. The particular allocation could well be an effect of random initialization. I could also see other reasons for this particular allocation than phrases being more useful in intermediate layers (e.g., positional encodings in the first layer is a strong bias towards token-to-token attention, it could be that the magnitude of convolved vectors is larger than the batch-normalized unigram encodings, so that logits are larger.
Questions
--------------
In "query-as-kernel convolution", it is unclear whether you map Q[t, :] into n x d_q x d_k convolution kernel parameters, or if each element of the window around Q[t] of width n is mapped to a convolution kernel parameter. Also what is the exact form of the transformation. Do you transform the d_q dimensional vectors in Q to a d_q x d_k matrix? Is this done by mapping to a d_q * d_k dimensional vector which is then rearranged into the convolution kernel matrix?
Does the model tend to choose one particular n-gram type for a particular position, or will it select different n-gram types for the same position?
"The selection of which n-gram to assign to how many heads is arbitrary" - How is this arbitrary? This seems a rather strong inductive bias?
"However, the homogeneity restriction may limit the model to learn interactions between different n-gram types" - How is the case? It seems rather that the limitation is that the model cannot dynamically allocate heads to the most relevant n-gram type?
I do not understand equation 14. Do you mean I_dec = I_cross = (...)?
"Phrase-to-phrase mapping helps model local agreement, e.g., between an adjective and a noun (in terms of gender, number and case) or between subject and verb (in terms of person and number)." Is this actually verified with experiments / model inspection?
"This is especially necessary when the target language is morphologically rich, like German, whose words are usually compounded with sub-words expressing different meanings and grammatical structures" This claim should be verified, e.g. by comparing to English-French as well as model inspection.