Recurrent neural networks (RNNs) are powerful models for sequential data. They can approximate arbitrary computations, and have been used successfully in domains such as text and speech. However, the flexibility of RNNs makes them susceptible to overfitting and regularization is important. We develop a noise-based regularization method for RNNs. The idea is simple and easy to implement: we inject noise in the hidden units of the RNN and then maximize the original RNN's likelihood averaged over the injected noise. On a language modeling benchmark, our method achieves better performance than its deterministic RNN and variational dropout (Gal and Ghahramani, 2016).
The RNN transition function is: h_t+1 = f(h_t,x_t)
This paper proposes using a stochastic transition function instead of a deterministic one.
i.e h_{t+1} \sim expfam(mean = f(h_t,x_t), gamma) where expfam denotes a distribution from the exponential family.
The experimental results consider text modeling (evaluating on perplexity) on Penn Treebank and Wikitext-2. The method of regularization is compared to a reimplementation of Variational Dropout and no regularization.
The work is written clearly and easy to follow.
Overall, the core idea in this work is interesting but underexplored. 
* As of when I read this paper, all results on this work used 200 hidden units realizing results that were well off from the state of the art results on Penn Tree Bank (as pointed out by the external reader).
The authors responded by stating that this was done to achieve a relative comparison.  A more interesting comparison, in addition to the ones presented, would be to see how well each method performs while not controlling for hidden layer size. Then, it might be that restricting the number of hidden dimensions is required for the RNN without any regularization but for both Variational Dropout and Noisin, one obtains better results with a larger the hidden dimension.
* The current experimental setup makes it difficult to assess when the proposed regularization is useful. Table 2 suggests the answer is sometimes and Table 3 suggests its marginally useful when the RNN size is restricted.
* How does the proposed method's peformance compare to Zoneout https://arxiv.org/pdf/1606.01305.pdf?
* Clarifying the role of variational inference: I could be missing something but I don't see a good reason why the prior (even if learned) should be close to the true posterior under the model. I fear the bound in Section (3) [please include equation numbers in the paper] could be quite loose.
* What is the rationale for not comparing to the model proposed in [Chung et. al] where there is a stochastic and deterministic component to the transition function? In what situations do we expect the fully stochastic transition here to work better than a model that has both? Presumably, some aspect of the latent variable + RNN model could be expressed by having a small variance for a subset of the dimensions and large one for the others
but since gamma is the same across all dimensions of the model, I'm not sure this feature can be incorporated into the current approach. Such a comparison would also empirically verify what happens when learning with the prior versus doing inference with an approximate posterior helps.
* The regularization is motivated from the point of view of sampling the hidden states to be from the exponential family, but all the experiments provided seem to use a Gaussian distribution. This paper would be strengthened by a discussion and experimentation with other kinds of distributions in the exponential family.