A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
Undirected neural sequence models such as BERT (Devlin et al., 2019) have received renewed interest due to their success on discriminative natural language understanding tasks such as question-answering and natural language inference. The problem of generating sequences directly from these models has received relatively little attention, in part because generating from such models departs significantly from the conventional approach of monotonic generation in directed sequence models. We investigate this problem by first proposing a generalized model of sequence generation that unifies decoding in directed and undirected models. The proposed framework models the process of generation rather than a resulting sequence, and under this framework, we derive various neural sequence models as special cases, such as autoregressive, semi-autoregressive, and refinement-based non-autoregressive models. This unification enables us to adapt decoding algorithms originally developed for directed sequence models to undirected models. We demonstrate this by evaluating various decoding strategies for a cross-lingual masked translation model (Lample and Conneau, 2019). Our experiments show that generation from undirected sequence models, under our framework, is competitive with the state of the art on WMT*14 English-German translation. We also demonstrate that the proposed approach enables constant-time translation with similar performance to linear-time translation from the same model by rescoring hypotheses with an autoregressive model.
This paper focuses on decoding/generation in neural sequence models (specifically machine translation) in a non-autoregressive manner that instead of generating in a left-to-right manner, focuses on generating sequences by picking a length first ,and then indices to replace in a deterministic or random scheme and, finally using a context sensitive distribution over vocabulary (using BERT-like masked LM scheme) to pick the word to replace. In practice, this procedure of picking indices and words to replace is repeated T number of times and hence the final sequence is obtained by this iterative refinement procedure. This is an interesting and important research direction because not only would it result in better and context sensitive greedy/approximate-MAP decoded outputs, but also opens up opportunities for parallelization of the decoding procedure which is difficult to achieve with left-to-right decoders. That said, the results are fairly inconclusive and the practical implementation does leave things desired for a practical decoder. As observed by the authors, different deterministic strategies for choosing T results in very different performances among the variants of the proposed approach. Besides among the variants, one clear pattern is that uniformly random picking of indices is worse than other schemes (left-to-right, least-to-most, easy-first) which is not unexpected but no conclusive empirical evidence can be found for relative differences between the performances of other 3 schemes. Moreover, the proposed decoding variants generally perform worse than or at best similarly to standard autoregressive baselines. As authors note, this is due to the mismatch between the method in which the model was trained and the decoding procedure which is not surprising, but at the same time this does not give insight into the effectiveness of the proposed decoding objective. The central question is: if the training prefers left-to-right generation then how valuable is it to device more reasonable but incompatible decoding procedures? Also, authors also note that index picking schemes investigated in the paper are heuristic based and a more interesting decoder could be learned if index selection procedure itself was learned with features depending on the previous index selection states, decoded states Y, and other relevant quantities. They attribute poor performance of the proposed decoder to the nature of index selection approaches investigated in the paper. I think the paper would be strengthened with results with a more sophisticated learned index selection procedure in addition to the heuristics investigated in this paper. Overall, while the idea and motivation behind this work is exciting, the inconclusive results and the approaches for practical implementation leave open significant room for improvement.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
Undirected neural sequence models such as BERT (Devlin et al., 2019) have received renewed interest due to their success on discriminative natural language understanding tasks such as question-answering and natural language inference. The problem of generating sequences directly from these models has received relatively little attention, in part because generating from such models departs significantly from the conventional approach of monotonic generation in directed sequence models. We investigate this problem by first proposing a generalized model of sequence generation that unifies decoding in directed and undirected models. The proposed framework models the process of generation rather than a resulting sequence, and under this framework, we derive various neural sequence models as special cases, such as autoregressive, semi-autoregressive, and refinement-based non-autoregressive models. This unification enables us to adapt decoding algorithms originally developed for directed sequence models to undirected models. We demonstrate this by evaluating various decoding strategies for a cross-lingual masked translation model (Lample and Conneau, 2019). Our experiments show that generation from undirected sequence models, under our framework, is competitive with the state of the art on WMT*14 English-German translation. We also demonstrate that the proposed approach enables constant-time translation with similar performance to linear-time translation from the same model by rescoring hypotheses with an autoregressive model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper focuses on decoding/generation in neural sequence models (specifically machine translation) in a non-autoregressive manner that instead of generating in a left-to-right manner, focuses on generating sequences by picking a length first ,and then indices to replace in a deterministic or random scheme and, finally using a context sensitive distribution over vocabulary (using BERT-like masked LM scheme) to pick the word to replace. In practice, this procedure of picking indices and words to replace is repeated T number of times and hence the final sequence is obtained by this iterative refinement procedure. This is an interesting and important research direction because not only would it result in better and context sensitive greedy/approximate-MAP decoded outputs, but also opens up opportunities for parallelization of the decoding procedure which is difficult to achieve with left-to-right decoders. That said, the results are fairly inconclusive and the practical implementation does leave things desired for a practical decoder. As observed by the authors, different deterministic strategies for choosing T results in very different performances among the variants of the proposed approach. Besides among the variants, one clear pattern is that uniformly random picking of indices is worse than other schemes (left-to-right, least-to-most, easy-first) which is not unexpected but no conclusive empirical evidence can be found for relative differences between the performances of other 3 schemes. Moreover, the proposed decoding variants generally perform worse than or at best similarly to standard autoregressive baselines. As authors note, this is due to the mismatch between the method in which the model was trained and the decoding procedure which is not surprising, but at the same time this does not give insight into the effectiveness of the proposed decoding objective. The central question is: if the training prefers left-to-right generation then how valuable is it to device more reasonable but incompatible decoding procedures? Also, authors also note that index picking schemes investigated in the paper are heuristic based and a more interesting decoder could be learned if index selection procedure itself was learned with features depending on the previous index selection states, decoded states Y, and other relevant quantities. They attribute poor performance of the proposed decoder to the nature of index selection approaches investigated in the paper. I think the paper would be strengthened with results with a more sophisticated learned index selection procedure in addition to the heuristics investigated in this paper. Overall, while the idea and motivation behind this work is exciting, the inconclusive results and the approaches for practical implementation leave open significant room for improvement.
assistant
This paper focuses on decoding/generation in neural sequence models (specifically machine translation) in a non-autoregressive manner that instead of generating in a left-to-right manner, focuses on generating sequences by picking a length first ,and then indices to replace in a deterministic or random scheme and, finally using a context sensitive distribution over vocabulary (using BERT-like masked LM scheme) to pick the word to replace. In practice, this procedure of picking indices and words to replace is repeated T number of times and hence the final sequence is obtained by this iterative refinement procedure. This is an interesting and important research direction because not only would it result in better and context sensitive greedy/approximate-MAP decoded outputs, but also opens up opportunities for parallelization of the decoding procedure which is difficult to achieve with left-to-right decoders. That said, the results are fairly inconclusive and the practical implementation does leave things desired for a practical decoder. As observed by the authors, different deterministic strategies for choosing T results in very different performances among the variants of the proposed approach. Besides among the variants, one clear pattern is that uniformly random picking of indices is worse than other schemes (left-to-right, least-to-most, easy-first) which is not unexpected but no conclusive empirical evidence can be found for relative differences between the performances of other 3 schemes. Moreover, the proposed decoding variants generally perform worse than or at best similarly to standard autoregressive baselines. As authors note, this is due to the mismatch between the method in which the model was trained and the decoding procedure which is not surprising, but at the same time this does not give insight into the effectiveness of the proposed decoding objective. The central question is: if the training prefers left-to-right generation then how valuable is it to device more reasonable but incompatible decoding procedures? Also, authors also note that index picking schemes investigated in the paper are heuristic based and a more interesting decoder could be learned if index selection procedure itself was learned with features depending on the previous index selection states, decoded states Y, and other relevant quantities. They attribute poor performance of the proposed decoder to the nature of index selection approaches investigated in the paper. I think the paper would be strengthened with results with a more sophisticated learned index selection procedure in addition to the heuristics investigated in this paper. Overall, while the idea and motivation behind this work is exciting, the inconclusive results and the approaches for practical implementation leave open significant room for improvement.