Published as a conference paper at ICLR 2018 REGULARIZING AND OPTIMIZING LSTM LANGUAGE MODELS
In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM, which uses DropConnect on hidden-tohidden weights, as a form of recurrent regularization. Further, we introduce NTAvSGD, a non-monotonically triggered (NT) variant of the averaged stochastic gradient method (AvSGD), wherein the averaging trigger is determined using a NT condition as opposed to being tuned by the user. Using these and other regularization strategies, our AvSGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2. We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart. The code for reproducing the results is open sourced and is available at https://github.com/salesforce/ awd-lstm-lm.
This is a well-written paper that proposes regularization and optimization strategies for word-based language modeling tasks.   The authors propose the use of DropConnect  on the hidden-hidden connections as a regularization method, in order to take advantage of high-speed LSTM implementations via the cuDNN LSTM libraries from NVIDIA.  The  focus of this work is on the prevention of overfitting on the recurrent connections of the LSTM.  The authors explore a variant of Average-SGD (NT-ASGD) as an optimization strategy which eliminates the need for tuning the average trigger and uses a constant learning rate.  Averaging is triggered when the validation loss worsens or stagnates for a few cycles, leading to two new hyper parameters: logging interval and non-monotone interval.  Other forms of well-know regularization methods were applied to the non-recurrent connections, input, output and embedding matrices.  
As the authors point out, all the methods used in this paper have been proposed before and theoretical convergence explained. The novelty of this work lies in its successful application to the language modeling task achieving state-of-the-art results.
On the PTB task, the proposed AWD-LSTM achieves a perplexity of 57.3 vs 58.3 (Melis et al 2017) and almost the same perplexity as Melis et el. on the Wiki-Text2 task (65.8 vs 65.9).  The addition of a cache model provides significant gains on both tasks.   
It would be useful, if authors had explored the behavior of the  AWD-LSTM algorithm with respect to various hyper parameters  and provided a few insights towards their choices for other large vocabulary language modeling tasks (1 million vocabulary sizes).  
Similarly, the choice of the average trigger and number of cycles seem arbitrary -  it would have been good to see a graph over a range of values, showing their impact on the model's performance.
A 3-layer LSTM has been used for the experiments  - how was this choice made?  What is the impact of this algorithm if the net was a 2-layer net as is typical in most large-scale LMs?
Table 3 is interesting to see how the cache model helps with rare words  and as such has applications in key word spotting tasks. Were the hyper parameters of the cache tuned to perform better on rare words?  More details on the design of the cache model would have been useful.
You state that the gains obtained using the cache model were far less than what was obtained in Graves et al 2016 - what do you attribute this to?
Ablation analysis in Table 4 is very useful - in particular it shows how lack of regularization of the recurrent connections can lead to maximum degradation in performance.
Most of the results in this paper have been based on one choice of various model parameters. Given the emperical nature of this work, it would have made the paper even clearer if an analysis of their choices were presented.  Overall, it would be beneficial to the MLP community to see this paper accepted in the conference.