Many regularization methods have been proposed to prevent overfitting in neural networks. Recently, a regularization method has been proposed to optimize the variational lower bound of the Information Bottleneck Lagrangian. However, this method cannot be generalized to regular neural network architectures. We present the activation norm penalty that is derived from the information bottleneck principle and is theoretically grounded in a variation dropout framework. Unlike in previous literature, it can be applied to any general neural network. We demonstrate that this penalty can give consistent improvements to different state of the art architectures both in language modeling and image classification. We present analyses on the properties of this penalty and compare it to other methods that also reduce mutual information.
This paper tries to create a mapping between activation norm penalties and information bottleneck framework using variational dropout framework. While I find the path taken interesting, the paper itself is hard to follow, mostly due to constantly flipping notation (cons section below lists some of the issues) and other linguistic errors. In the current form, this work is somewhere between a theoretical paper and an empirical one, however for a theoretical one it lacks strictness, while for empirical one - novelty.
From theoretical perspective:
The main claim in this paper seems to be (10), however it is not formalised in any form of theorem, and so -- lacks a lock of strictness. Even under the assumption that it is updated, and made more strict - what is a crucial problem is a claim, that after arriving at:
tr[ K(X, X) ] - ln( det[ K(X, X) ] )
dropping the log determinant is anyhow justified, to keep the reasoning/derivation of the whole method sound. Authors claim that quote "As for the determinant of the covariance matrix of Gaussian Process, we cannot easily evaluate or
derive its gradient, so we do not include it in our computation." Which is not a justification for treating the derivation as a proper connection between penalising activities norm and information bottleneck idea. Terms like this will emerge in many other models, where one assumes diagonal covariance Gaussians; in fact the easiest model to justify this penalty is just to say one introduces diagonal Gaussian prior over activations, and that's it. Well justified penalty, easy to connect to many generalisation bound claims. However in the current approach the connection is simply not proven in the paper. 
From practical perspective:
Activation norm penalties are well known objects, used for many years (L1 activation penalty for at least 6 years now, see "Deep Sparse Rectiﬁer Neural Networks"; various activation penalties, including L2, changes in L2, etc. in Krueger PhD dissertation). Consequently for a strong empirical paper I would expect much more baselines, including these proposed by Krueger et al.
Pros:
- empirical improvements shown on two different classes of problems.
- interesting path through variational dropout is taken to show some equivalences
Cons:
- there is no proper proof of claimed connection between IB and ANP, as it would require a determinant of K(X, X) to be 1.
- work is not strict enough for a theoretical paper, and does not include enough comparison for empirical one
- paper is full of typing/formatting/math errors/not well explained objects, which make it hard to read, to name a few:
 * fonts of objects used in equations change through the text - there is a \textbf{W} and normal W, \textbf{I} and normal I, similarly with X, Ts etc. without any explanation. I am assuming font are assigned randomly and they represent the same object.
 * dydp in (5) is in the wrong integral
* \sigma switches meaning between non-linearity and variance
* what does it mean to define a normal distribution with 0 variance (like in (1)). Did authors mean  an actual "degenerate Gaussian", which does not have a PDF? But then p(y|t) is used in (5), and under such definition it does not exist, only CDF does. 
* \Sigma_1 in 3.2 is undefined, was r(t) supposed to be following N(0, \Sigma_1)  instead of written N(0, I)?