A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.
In the new dropout variant, we repeat the same dropout mask at each time step for both inputs, outputs, and recurrent layers (drop the same network units at each time step). This is in contrast to the existing ad hoc techniques where different dropout masks are sampled at each time step for the inputs and outputs alone (no dropout is used with the recurrent connections since the use of different masks with these connections leads to deteriorated performance). Our method and its relation to existing techniques is depicted in figure 1. When used with discrete inputs (i.e. words) we place a distribution over the word embeddings as well. Dropout in the word-based model corresponds then to randomly dropping word types in the sentence, and might be interpreted as forcing the model not to rely on single words for its task.
We next survey related literature and background material, and then formalise our approximate inference for the Variational RNN, resulting in the dropout variant proposed above. Experimental results are presented thereafter.
This paper proposes a novel approach of dropout for Recurrent Neural Networks (RNN). RNN in this paper is formulated as a Bayesian Neural Network. The authors use Variational Inference to train the RNN. They use Gaussian distributions as variational distributions for weight parameters. A novel dropout can be applied to this approach by using 0 mean Gaussian distribution as a part of variational distributions.  Previous empirical works suggested that dropout over recurrent layer was not effective. This paper suggests that bayesian approach is effective for dropout over recurrent layer. In addition, the authors apply the novel dropout to weight parameters for word embedding.  Experimental results show that this approach is effective to avoid overfitting.
Line 205: Previous method has 0 for weight decay. However, authors tune the weight decay for their proposed method. Does the proposed method outperform the previous method when the weight decay has 0? Tuning weight decay may have an enormous influence on the experimental results. I think that experimental conditions should be same except for the part of the dropout that authors propose.  Line 287-288: Why does test error decrease and then increase before decreasing again?  Figure4(a): When P_U = 0.5 and P_E = 0.0, test error can decrease after the number of epochs is 10^3. The authors should show the test error after the number of epochs is 10^3.  Figure 4(b): Why is the number of epochs less than other experiments such as Figure 4(a) and 4(c)? Figure 4(a) shows that test error drastically change after the number of epochs is 10^2. Figure 4(b) should show test error after the number of epochs is 10^2.  Figure 4(b) suggests that dropout for recurrent layer gives high test error. I think that dropout for recurrent layer is not effective even if the proposed method is applied.  In addition, Figure 4(a) and 4(b) suggest that dropout for embedding layer is more important than dropout for recurrent layer.  Other: Authors should unify the description when they refer figures in this paper. For example, “fig” is used in Line 272 but “figure”  is used in Line 285.