Non-autoregressive models are promising on various text generation tasks. Previous work hardly considers to explicitly model the positions of generated words. However, the position modeling is an essential problem in non-autoregressive text generation. In this study, we propose PNAT, which incorporates positions as a latent variable into the text generative process. Experimental results show that PNAT achieves top results on machine translation and paraphrase generation tasks, outperforming several strong baselines.
This work proposes an alternative approach to non-autoregressive translation (NAT) by predicting positions in addition to the word identities, such that the word order in the final prediction doesn't matter as long as the positions are correct. The length of the translation is predicted similar to Gu et al 2017, as well as smoothly copying the source sequence to decoder input. However, since the positions are unknown, this paper employs a heuristic search method to find the nearest neighbors in the embedding space to obtain position supervision.
Experiments are conducted on a machine translation task and a paraphrase generation task. For WMT14 DE-EN and IWSLT16 DE-EN, this method gets superior performance compared to baselines. For paraphrase generation, this approach with an AR position predictor gets even better performance than a normal AR transformer. Further analysis shows that this approach degrades less without knowledge distillation, or without removing repetitions.
Pros:
1. This approach gets better performance compared to baselines.
2. The idea of modeling positions as a latent variable is interesting and might generalize to other tasks beyond NAT.
Cons:
1. This work should compare to later baselines such as FlowSeq (https://arxiv.org/pdf/1909.02480.pdf) which gets better performance with flow.
2. In table 1, although the proposed approach outperforms imitate-NAT, the speedup is lower, making it hard to judge which is actually better.
3. In table 2, why is AR predictor used? What's the performance of NAR predictor? (in general why consider NAR position predictor at all?)
4. It is not clear why heuristic search would work here. Is any pretraining required? Otherwise, since there's no gradient signal for the positions, I'm not sure how the model figures it out.
Questions:
1. How many samples are used in table 1 LPD? Or is it argmax decoding for each length?
2. Is it possible to include a few examples showing predicted positions in an appendix to help better understand the model's behavior?
3. Why do you think positions can be predicted in a NAR manner? Isn't it just shifting the burdens to the position predictor? (Since in transformers if it's able to learn positions then it should be trivial to reorder based on those positions)
Minor details:
Some typos need to be fixed.
Overall, it is an interesting idea to predict the positions and word identities separately, and with Gumbel-Softmax and VAE we might be able to optimize the true marginals instead of relying on heuristic search. However, empirically there's been better performance achieved by flow-based models, so I am inclined to reject this work.
--------updates after reading the authors' rebuttal-----
I really appreciate the substantial experiments the authors conducted post-review. However, I still have some concerns regarding the baseline flowseq. Flowseq-large gets better results but is not reported in the main paper. Besides, it is unclear how is the speedup computed since Ma et al 2019 reported that the speedup depends on batch size, and at batch size 120 the speedup is ~X6 for flowseq-base.
Minor issue: in table 1 which dataset are the speedup numbers based on?