We propose Noisy Information Bottlenecks (NIB) to limit mutual information between learned parameters and the data through noise. We show why this benefits generalization and allows mitigation of model overfitting both for supervised and unsupervised learning, even for arbitrarily complex architectures. We reinterpret methods including the Variational Autoencoder, β-VAE, network weight uncertainty and a variant of dropout combined with weight decay as special cases of our approach, explaining and quantifying regularizing properties and vulnerabilities within information theory.
This paper studies "Noisy Information Bottlenecks". The overall idea is that, if the mutual information between learned parameters and the data is limited, then this prevents overfitting. It proposes to create a "bottleneck" to limit the mutual information. Specifically, the bottleneck is created by having the data depend on a noisy version of the parameters, rather than the true parameters and invoking the information processing inequality. The paper gives an example of Gaussian mean field inference. Ultimately, the analysis boils down to looking at a signal-to-noise ratio of the algorithm, which looks very much like regularization.
I think this is a very interesting direction, but the present paper is somewhat unclear. In particular, the example in section 3.1 says that a noisy information bottleneck is introduced, but then says that the modified and unmodified models have "training algorithms that are exactly equivalent." I think this example needs to be clarified. Many of the parameters here are also unclear and not properly defined/introduced. What is the relationship between $\theta$ and $\tilde\theta$ exactly? In this simple model, can we not calculate the mutual information directly (i.e., without the bottleneck)?
The connection between mutual information and generalization has been studied in several contexts [see, e.g., the references in this paper and https://arxiv.org/abs/1511.05219 https://arxiv.org/abs/1705.07809 https://arxiv.org/abs/1712.07196 https://arxiv.org/pdf/1605.02277.pdf https://arxiv.org/abs/1710.05233 https://arxiv.org/pdf/1706.00820.pdf ] and further exploration is desirable. This paper is giving an information-theoretic perspective on existing variational inference methods. Such a perspective is interesting, but needs to be further developed and explained. Specifically, how can mutual information in this context be formally linked to generalization/overfitting? Also, the definition of mutual information used in this paper uses the inferred distribution q (e.g., in eq. 2), which is somewhat unusual. As a result, constraining the model will alter the mutual information and I think the effect of this should be remarked on.
Overall, I think this paper has some interesting ideas, but those need to be fleshed out and clearly explained in a future revision.