VARIATIONAL SMOOTHING IN RECURRENT NEURAL NETWORK LANGUAGE MODELS
We present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017). We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e., a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods.
This submission closely builds upon an earlier work (Xie et al., 2017, Gal and Ghahramani, 2016) and proposes a new data noising technique motivated by Bayesian RNNs. Specifically, the key contribution is to extend Gal and Ghahramani (2016) to word embedding noising, while drawing inspiration from trational data smoothing techniques. Some variants are discussed, including those motivated by linear interpolation and Kneser-Ney smoothing, just as in Xie et al., 2017. Empirical evaluation is performed with language modeling experiments, and the proposed methods outperforms comparable baselines. One can imagine such a technique could be useful in many other sequence tasks. 
The paper is well-motivated and clearly written, and the experiments seem reasonable to me. Therefore I would vote for acceptance. My concern, which is not major, is that the proposed method might be a bit incremental based on Xie et al. (2017) and Gal and Ghahramani (2016).
Pros:
- Theoratical justification seems reasonably sound to me.
- Strong empirical performance.
Cons: 
- It would be interesting to see how the proposed technique works when applied to state-of-the-art models.
Details:
- I'm not entirely familiar with Gal and Ghahramani (2016), but I'm assuming the discussion in Section 3 and how it extends to word embeddings are from this earlier work. Please correct me if I'm wrong, so that I can adjust my recommendation accordingly.
- I can't find anything describing how \sigma is determined.
- Page 4, the paragraph of `Training.` I can't parse `we go though sequence t multiple times`.
- \ell_2 Regularization for `VS` models. I'm confused here, isn't the coefficient for VS determined by Eq. 3? Why is it still tuned in Section 5.1?
- Elementwise smoothing: I'm confused why one needs to sample \alpha for each dimension. Can't it be done by sampling a mask, just as in dropout?