Machine translation is an important real-world application, and neural networkbased AutoRegressive Translation (ART) models have achieved very promising accuracy. Due to the unparallelizable nature of the autoregressive factorization, ART models have to generate tokens one by one during decoding and thus suffer from high inference latency. Recently, Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time. However, they could only achieve inferior accuracy compared with ART models. To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a welltrained ART model to train the NART model. We define two hints for the machine translation task: hints from hidden states and hints from word alignments, and use such hints to regularize the optimization of NART models. Experimental results show that the NART model trained with hints could achieve significantly better translation performance than previous NART models on several tasks. In particular, for the WMT14 En-De and De-En task, we obtain BLEU scores of 25.20 and 29.52 respectively, which largely outperforms the previous non-autoregressive baselines. It is even comparable to a strong LSTM-based ART model (24.60 on WMT14 En-De), but one order of magnitude faster in inference.
This work proposes a non-autoregressive Neural Machine Translation model which the authors call NART, as opposed to an autoregressive model which is referred to as an ART model. The main idea behind this work is to leverage a well trained ART model to inform the hidden states and the word alignment of NART models. The joint distribution of the targets y given the inputs x, is factorized into two components as in previous works on non-autoregressive MT: an intermediate z which is first predicted from x, which captures the autoregressive part, while the prediction of y given z is non-autoregressive. This is the approach taken e.g., in Gu et al, Kaiser et al, Roy et al., and this also seems to be the approach of this work.  The authors argue that improving the expressiveness of z (as was done in Kaiser et al, Roy et al), is expensive and so the authors propose a simple formulation for z. In particular, z is a sequence of the same length as the targets, where the j^{th} entry z_j is a weighted sum of the embedding of the inputs x (the weights depend in a deterministic fashion on j) . Given this z, the model predicts the targets completely non-autoregressively. However, this by itself is not entirely sufficient, and so the authors also utilize "hints": 1) If the pairwise cosine similarity between two successive hidden states in the student  NART model is above a certain threshold, while the similarity is lower than another threshold in the ART model, then the NART model incurs a cost proportional to this similarity 2) A KL term is used to encourage the distribution of attention weights of the student ART model to match that of the teacher NART model. These two loss terms are used in different proportions (using additional hyperparameters) together with maximizing the likelihood term.
Quality: The paper is not very well written and is often hard to follow in parts. Here are some examples of the writing that feel awkward:
--  Consequently, people start to develop Non-AutoRegressive neural machine
Translation (NART) models to speed up the inference process (Gu et al., 2017; Kaiser et al., 2018;
Lee et al., 2018). 
-- In order to speed up to the inference process, a line of works begin to develop non-autoregressive
translation models. 
Originality: The idea of using an autoregressive teacher model to improve a non-autoregressive translation model has been used in Gu et al., Roy et al., where knowledge distillation is used. So knowledge distillation paper from Hinton et al., should be cited. Moreover, the authors have missed comparing their work to that of Roy et al. (https://arxiv.org/abs/1805.11063), which greatly improves on the work of Kaiser et al., and almost closes the gap between a non-autoregressive model and an autoregressive model (26.7 BLEU vs 27 BLEU on En-De) while being orders of magnitude faster. So it is not true that:
-- "While the NART models achieve significant speedup during inference (Gu et al., 2017), their accuracy
is considerably lower than their ART counterpart."
-- "Non-autoregressive translation (NART) models have suffered from low-quality translation results"
Significance: The work introduces the idea of using hints for non-autoregressive machine translation. However, I have a technical concern: It seems that the authors complain that previous works like Kaiser et al, Roy et al, use sophisticated submodules to help the expressiveness of z and this was the cause for slowness. However, the way the authors define z seems to have some problems:
- z_j does not depend on z_1, ..., z_{j-1}, so where is the autoregressive dependencies being captured?
- z_1, z_2, ..., z_{T_y} depends only on the length of y, and does not depend on y in any other way. Given x, predicting z is trivial and I don't see why that should help the model f(y | z, x) help at all? 
- Given such a trivial z, one can just assume that your model is completely factorial i.e. P(y|x) = \prod_{i} P(y_i|x) since the intermediate z has no information on the y's except it's length.
This is quite suspicious to me, and it seems that if this works, then a completely factorial model should work as well if we only use the "hints" from the ART teacher model. This is a red flag to me, and I am finding this hard to believe.