NON-AUTOREGRESSIVE NEURAL MACHINE TRANSLATION
Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English-German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 EnglishRomanian.
This work proposes non-autoregressive decoder for the encoder-decoder framework in which the decision of generating a word does not depends on the prior decision of generated words. The key idea is to model the fertility of each word so that copies for source words are fed as input to the encoder part, not the generated target words as inputs. To achieve the goal, authors investigated various techniques: For inference, sample fertility space for generating multiple possible translations. For training, apply knowledge distilation for better training followed by fine tuning by reinforce. Experiments for English/German and English/Romanian show comparable translation qualities with speedup by non-autoregressive decoding.
The motivation is clear and proposed methods are very sound. Experiments are carried out very carefully.
I have only minor concerns to this paper:
- The experiments are designed to achieve comparable BLEU with improved latency. I'd like to know whether any BLUE improvement might be possible under similar latency, for instance, by increasing the model size given that inference is already  fast enough.
- It'd also like to see other language pairs with distorted word alignment, e.g., Chinese/English, to further strengthen this work, though  it might have little impact given that attention already capture sort of alignment.
- What is the impact of the external word aligner quality? For instance, it would be possible to introduce a noise in the word alignment results or use smaller data to train a model for word aligner. 
- The positional attention is rather unclear and it would be better to revise it. Note that equation 4 is simply mentioning attention computation, not the proposed positional attention.