Regularizing and Optimizing LSTM Language Models
In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM, which uses DropConnect on hidden-to-hidden weights, as a form of recurrent regularization. Further, we introduce NT-ASGD, a non-monotonically triggered (NT) variant of the averaged stochastic gradient method (ASGD), wherein the averaging trigger is determined using a NT condition as opposed to being tuned by the user. Using these and other regularization strategies, our ASGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2. We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart. The code for reproducing the results is open sourced and is available at https://github.com/salesforce/awd-lstm-lm.
 The paper sets a new state of the art on word level language modelling on the Penn Treebank and Wikitext-2 datasets using various optimization and regularization techniques. These already very good results are further improved, by a large margin, using a Neural Cache. The paper is well written, easy to follow and the results speak for themselves. One possible criticism is that the experimental methodology does not allow for reliable conclusions to be drawn about contributions of all different techniques, because they seem to have been evaluated at a single hyperparameter setting (that was hand tuned for the full model?). A variant on the Averaged SGD method is proposed. This so called NT-ASGD optimizer switches to averaging mode based on recent validation losses. I would have liked to see a more thorough assessment of NT-ASGD, especially against well tuned SGD. I particularly liked Figure 3 which shows how the Neural Cache makes the model much better at handling rare words and UNK (!) at the expense of very common words. Speaking of the Neural Cache, a natural baseline would have been dynamic evaluation. All in all, the paper is a solid contribution which deserves to be accepted. It could become even better, were the experiments to tease the various factors apart.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Regularizing and Optimizing LSTM Language Models
In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM, which uses DropConnect on hidden-to-hidden weights, as a form of recurrent regularization. Further, we introduce NT-ASGD, a non-monotonically triggered (NT) variant of the averaged stochastic gradient method (ASGD), wherein the averaging trigger is determined using a NT condition as opposed to being tuned by the user. Using these and other regularization strategies, our ASGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2. We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart. The code for reproducing the results is open sourced and is available at https://github.com/salesforce/awd-lstm-lm.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper sets a new state of the art on word level language modelling on the Penn Treebank and Wikitext-2 datasets using various optimization and regularization techniques. These already very good results are further improved, by a large margin, using a Neural Cache. The paper is well written, easy to follow and the results speak for themselves. One possible criticism is that the experimental methodology does not allow for reliable conclusions to be drawn about contributions of all different techniques, because they seem to have been evaluated at a single hyperparameter setting (that was hand tuned for the full model?). A variant on the Averaged SGD method is proposed. This so called NT-ASGD optimizer switches to averaging mode based on recent validation losses. I would have liked to see a more thorough assessment of NT-ASGD, especially against well tuned SGD. I particularly liked Figure 3 which shows how the Neural Cache makes the model much better at handling rare words and UNK (!) at the expense of very common words. Speaking of the Neural Cache, a natural baseline would have been dynamic evaluation. All in all, the paper is a solid contribution which deserves to be accepted. It could become even better, were the experiments to tease the various factors apart.
assistant
 The paper sets a new state of the art on word level language modelling on the Penn Treebank and Wikitext-2 datasets using various optimization and regularization techniques. These already very good results are further improved, by a large margin, using a Neural Cache. The paper is well written, easy to follow and the results speak for themselves. One possible criticism is that the experimental methodology does not allow for reliable conclusions to be drawn about contributions of all different techniques, because they seem to have been evaluated at a single hyperparameter setting (that was hand tuned for the full model?). A variant on the Averaged SGD method is proposed. This so called NT-ASGD optimizer switches to averaging mode based on recent validation losses. I would have liked to see a more thorough assessment of NT-ASGD, especially against well tuned SGD. I particularly liked Figure 3 which shows how the Neural Cache makes the model much better at handling rare words and UNK (!) at the expense of very common words. Speaking of the Neural Cache, a natural baseline would have been dynamic evaluation. All in all, the paper is a solid contribution which deserves to be accepted. It could become even better, were the experiments to tease the various factors apart.