State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15 − 40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task.
The paper presentes a small extension to the Neural Transformer model of Vaswani et al 2017:
the multi-head attention computation (eq. 2,3):
head_i = Attention_i(Q,K,W)
MultiHead = Concat_i(head_i) * W = \sum_i head_i * W_i
is replaced with the so-called BranchedAttention (eq. 5,6,7,4):
head_i = Attention_i(Q,K,W)       // same as in the base model
BranchedAttention = \sum_i \alpha_i max(0, head_i * W_i * kappa_i * W^1 + b^1) W^2 + b^2
The main difference is that the results of application of each attention head is post-processed with a 2-layer ReLU network before being summed into the aggregated attention vector.
My main problem with the paper is understanding what really is implemented: the paper states that with alpha_i=1 and kappa_i=1 the two attention mechanism are equivalent. The equations, however, tell a different story: the original MultiHead attention quickly aggregates all attention heads, while the proposed BranchedAttention adds another processing step, effectively adding depth to the model.
Since the BranchedAttention is the key novelty of the paper, I am confused by this contradiction and treat it as a fatal flaw of this paper (I am willing to revise my score if the authors explain the equations) - the proposed attention either adds a small amount of parameters (the alphas and kappas) that can be absorbed by the other weights of the network, and the added alphas and kappas are easier/faster to optimize, as the authors state in the text, or the BranchedAttention works as shown in the equations, and effectively adds depth to the network by processing each attention's result with a small MLP before combining multiple attention heads. This has to be clarified before the paper is published.
The experiment show that the proposed change speeds convergence and improves the results by about 1 BLEU point. However, this requires a different learning rate schedule for the introduced parameters and some non-standard tricks, such as freezing the alphas and kappas during the end of the training.
I also have a questions about the presented results:
1) The numbers for the original transformer match the ones in Vaswani et al 2017, am I correct to assume that the authors did not rerun the tensor2tensor code and simply copied them from the paper?
2) Is all of the experimental setup the same as in Vaswani et al 2017? Are the results obtained using their tensor2tensor implementation, or are some hyperparameters different?
Detailed review:
Quality:
The equations and text in the paper contradict each other.
Clarity:
The language is clear, but the main contribution could be better explained.
Originality:
The proposed change is a small extension to the Neural Transformer model.
Significance:
Rather small, the proposed addition adds little modeling power to the network and its advantage may vanish with more data/different learning rate schedule.
Pros and cons:
+ the proposed approach is a simple way to improve the performance of multihead attentional models.
- it is not clear from the paper how the proposed extension works: does it regularize the model or dies it increase its capacity?