Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention. We also introduce two novel and interpretable approaches for latency control that are specifically designed for multiple attention heads. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach. We analyze how the latency controls affect the attention span and we study the relationship between the speed of a head and the layer it belongs to. Finally, we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.
The paper proposes an approach for simultaneous neural machine translation. While prior works deal with recurrent models, the authors adopt previous approaches for Transformer. Specifically, for decoder-encoder attention they introduce Monotonic Multihead Attention (MMA) designed to deal with several attention heads. Each attention head operates as a separate monotonic attention head similar to the ones from previous works by Raffel et al. (2017) and Arivazhagan et al. (2019). MMA has two versions: MMA-H (hard), where each head attends only to one token, and MMA-IL (infinite lookback), where each head attends to all tokens up to a selected position. 
The main novelty of the work in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions (L_var).
Experimental results on two translation tasks (IWSLT En-Vi and WMT De-En) show better quality-latency trade-offs compared to the recurrent analogs. Experiments with different number of attention heads would be helpful, but in the current state are rather confusing (see the comments below).
I can not recommend accepting this paper due to the two main reasons.
1) The proposed solution lacks novelty. 
MMA-H and MMA-IL attentions, introduced in this work, are straightforward applications of previous works by Raffel et al. (2017) and Arivazhagan et al. (2019) respectively to each of the heads. Novelty is in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions. However, these alone are unlikely to make a large impact.
2) The results are experimentally weak, evaluation is questionable. 
The current baseline is the recurrent model, but since the main contribution of this work is in how to deal with several heads the proper baselines would be (i) the same, but with single-head attention, (ii) the same, but without L_var (summing latency penalty over heads is straightforward). However, these baselines are absent and the improvement is likely to be due to the replacement of RNN with the Transformer - this would be a limited contribution.
Other comments on the experiments.
1) MMA-IL was tested only on one small dataset (IWSLT En-Vi), MMA-H only on 2 datasets - more experiments would help.
2) Figure 2 shows that when increasing lagging, performance first improves, then drops. While the former is expected, the latter is not: one would expect a model to reach the performance of its offline version when given maximum latency, but not to drop. The MILK model (Arivazhagan et al., 2019) behaves as expected, but the proposed MMA does not: does this mean that there is some problem with the model? An explanation of this behavior would be helpful.
3) Number of heads, Figure 4: the results are supposed to show that performance improves when increasing the number of heads, but the figure shows the opposite. If the decoder has more than one layer (which Transformer usually has), for 4 heads perform worse than one, 8 heads (which the standard Transformer setting) - not better than one, and we get the improvement only when using 16 heads with 6 layers. Could you elaborate on this? The text says “quality generally tends to improve with more layers and more heads”.
4) I can see the motivation for using the L_var loss term in this setting, but forcing heads in MMA-H attend to similar positions seems counterintuitive: there is evidence that making attention heads less correlated improves quality (see for example Li et al, EMNLP 2018 https://www.aclweb.org/anthology/D18-1317/), but L_var may end up doing the opposite. How different is the behavior of the learned heads? Figure 3 shows the average attention span, but this does not tell how many of the heads are doing different things. Some analysis would be nice.
Comments on the presentation.
1) It would be better to mention that encoder self-attention is limited only to previous positions earlier in the text and more prominently (now it’s on the bottom of the sixth page). A good place would be in Section 2.2 when talking about different attention types in the Transformer and modifications required for simultaneous translation.
2) Figure 2: it would be helpful to see offline model performance on these pictures (e.g., a horizontal line showing the BLEU scores). One of the main points of these experiments is to see how much each model drops compared to its offline version.
3) Figure 4: it is not clear from the figure which models it corresponds (for example, what is the difference between the first and the second figures: language pairs? models?)
4) Table 2 shows the BLEU scores for Transformer with unidirectional encoder and greedy decoding. It would be helpful to see also the BLEU scores of Transformer in the standard setting (full attention in the encoder, beam search inference). For now, it is not clear how much one loses by replacing the encoder to unidirectional: if much, then probably it makes sense to work also on the encoder (read and encode source sentence with a latency).
Language and typos.
The text is quite raw and could use an extra work; a lot of typos starting from the abstract (e.g., “...for multiple attentions heads”).