Decoding As Dynamic Programming For Recurrent Autoregressive Models
Decoding in autoregressive models (ARMs) consists of searching for a high scoring output sequence under the trained model. Standard decoding methods, based on unidirectional greedy algorithm or beam search, are suboptimal due to error propagation and myopic decisions which do not account for future steps in the generation process. In this paper we present a novel decoding approach based on the method of auxiliary coordinates (Carreira-Perpinan & Wang, 2014) to address the aforementioned shortcomings. Our method introduces discrete variables for output tokens, and auxiliary continuous variables representing the states of the underlying ARM. The auxiliary variables lead to a factor graph approximation of the ARM, whose maximum a posteriori (MAP) inference is found exactly using dynamic programming. The MAP inference is then used to recreate an improved factor graph approximation of the ARM via updated auxiliary variables. We then extend our approach to decode in an ensemble of ARMs, possibly with different generation orders, which is out of reach for the standard unidirectional decoding algorithms. Experiments on the text infilling task over SWAG and Daily Dialogue datasets show that our decoding method is superior to strong unidirectional decoding baselines.
This work introduces a new algorithm to improve decoding in discrete autoregressive models (ARMs). Because exact decoding of ARMs is computationally difficult, the contribution of this paper is to introduce a new approximate solution. The authors show experimentally that this new framework is very effective and outperforms multiple baselines such as greedy and bean search and another text filing algorithm called TIGC on a text-filling benchmark. This work is not in my domain of expertise so I am not able to have a very accurate evaluation. However, based on the references cited in this paper, there are other approximate solution for ARMs and I believe the authors need to use those as baselines to show that the proposed approximate solution is useful.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Decoding As Dynamic Programming For Recurrent Autoregressive Models
Decoding in autoregressive models (ARMs) consists of searching for a high scoring output sequence under the trained model. Standard decoding methods, based on unidirectional greedy algorithm or beam search, are suboptimal due to error propagation and myopic decisions which do not account for future steps in the generation process. In this paper we present a novel decoding approach based on the method of auxiliary coordinates (Carreira-Perpinan & Wang, 2014) to address the aforementioned shortcomings. Our method introduces discrete variables for output tokens, and auxiliary continuous variables representing the states of the underlying ARM. The auxiliary variables lead to a factor graph approximation of the ARM, whose maximum a posteriori (MAP) inference is found exactly using dynamic programming. The MAP inference is then used to recreate an improved factor graph approximation of the ARM via updated auxiliary variables. We then extend our approach to decode in an ensemble of ARMs, possibly with different generation orders, which is out of reach for the standard unidirectional decoding algorithms. Experiments on the text infilling task over SWAG and Daily Dialogue datasets show that our decoding method is superior to strong unidirectional decoding baselines.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work introduces a new algorithm to improve decoding in discrete autoregressive models (ARMs). Because exact decoding of ARMs is computationally difficult, the contribution of this paper is to introduce a new approximate solution. The authors show experimentally that this new framework is very effective and outperforms multiple baselines such as greedy and bean search and another text filing algorithm called TIGC on a text-filling benchmark. This work is not in my domain of expertise so I am not able to have a very accurate evaluation. However, based on the references cited in this paper, there are other approximate solution for ARMs and I believe the authors need to use those as baselines to show that the proposed approximate solution is useful.
assistant
This work introduces a new algorithm to improve decoding in discrete autoregressive models (ARMs). Because exact decoding of ARMs is computationally difficult, the contribution of this paper is to introduce a new approximate solution. The authors show experimentally that this new framework is very effective and outperforms multiple baselines such as greedy and bean search and another text filing algorithm called TIGC on a text-filling benchmark. This work is not in my domain of expertise so I am not able to have a very accurate evaluation. However, based on the references cited in this paper, there are other approximate solution for ARMs and I believe the authors need to use those as baselines to show that the proposed approximate solution is useful.