Entropy and mutual information in models of deep neural networks
We examine a class of stochastic deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physics methods, under the assumption that weight matrices are independent and orthogonally-invariant. (ii) We extend particular cases in which this result is known to be rigorously exact by providing a proof for two-layers networks with Gaussian random weights, using the recently introduced adaptive interpolation method. (iii) We propose an experiment framework with generative models of synthetic datasets, on which we train deep neural networks with a weight constraint designed so that the assumption in (i) is verified during learning. We study the behavior of entropies and mutual informations throughout learning and conclude that, in the proposed setting, the relationship between compression and generalization remains elusive.
The successes of deep learning methods have spurred efforts towards quantitative modeling of the performance of deep neural networks. In particular, an information-theoretic approach linking generalization capabilities to compression has been receiving increasing interest. The intuition behind the study of mutual informations in latent variable models dates back to the information bottleneck (IB) theory of [1]. Although recently reformulated in the context of deep learning [2], verifying its relevance in practice requires the computation of mutual informations for high-dimensional variables, a notoriously hard problem. Thus, pioneering works in this direction focused either on small network models with discrete (continuous, eventually binned) activations [3], or on linear networks [4,5].
In the present paper we follow a different direction, and build on recent results from statistical physics [6,7] and information theory [8,9] to propose, in Section 1, a formula to compute informationtheoretic quantities for a class of deep neural network models. The models we approach, described in Section 2, are non-linear feed-forward neural networks trained on synthetic datasets with constrained weights. Such networks capture some of the key properties of the deep learning setting that are usually difficult to include in tractable frameworks: non-linearities, arbitrary large width and depth, and correlations in the input data. We demonstrate the proposed method in a series of numerical experiments in Section 3. First observations suggest a rather complex picture, where the role of compression in the generalization ability of deep neural networks is yet to be elucidated.
The successes of deep learning methods have spurred efforts towards quantitative modeling of the performance of deep neural networks. In particular, an information-theoretic approach linking generalization capabilities to compression has been receiving increasing interest. The intuition behind the study of mutual informations in latent variable models dates back to the information bottleneck (IB) theory of [1]. Although recently reformulated in the context of deep learning [2], verifying its relevance in practice requires the computation of mutual informations for high-dimensional variables, a notoriously hard problem. Thus, pioneering works in this direction focused either on small network models with discrete (continuous, eventually binned) activations [3], or on linear networks [4,5].
In the present paper we follow a different direction, and build on recent results from statistical physics [6,7] and information theory [8,9] to propose, in Section 1, a formula to compute informationtheoretic quantities for a class of deep neural network models. The models we approach, described in Section 2, are non-linear feed-forward neural networks trained on synthetic datasets with constrained weights. Such networks capture some of the key properties of the deep learning setting that are usually difficult to include in tractable frameworks: non-linearities, arbitrary large width and depth, and correlations in the input data. We demonstrate the proposed method in a series of numerical experiments in Section 3. First observations suggest a rather complex picture, where the role of compression in the generalization ability of deep neural networks is yet to be elucidated.
Contributions of the paper:  The authors consider a stylized statistical model for data that respects neural network architecture, i.e. a Markov structure of the type T_\ell = \varphi(W_\ell*T_{\ell-1}, \xi_\ell ) where T_0 = X is the input, T_L = y is the output label, $W_\ell$ are random, independent weight matrices, \varphi is a nonlinearity applied elementwise on its first argument, possibly using  external randomness \xi_\ell. For data generated from this specific model, they make the following contributions.   1. They show that under this stylized model, one can obtain a simple formula for the (normalized i.e. per unit) entropy  H(T_\ell)/n and mutual information I(T_\ell; X)/n between the input data and each successive layer, in the high-dimensional limit. This formula is, in general, derived using the non-rigorous  replica method from statistical physics.   2. For a two-layer network and a class of nonlinearities, they rigorously establish this formula, using a technique of `adaptive' interpolation.   3. Based on this model, they provide some experimental results for two synthetic scenarios: the first based on the teacher-student model,  and the second for generative models, such as variational autoencoders. The experimental results are multi-faceted and include a comparison with entropy/mutual information estimators, validation of the replica formula, and some applications to the recent information bottleneck proposal of Tishby et al.    Summary:  The paper is a solid contribution, and I would argue that it is a clear accept. It would make a good addition to the NIPS program  and is of potential interest to communities possibly unfamiliar with the approaches in the paper.   Comments for the authors:  There are multiple aims that you have in the experiments section: which point to interesting applications for your results, like comparing entropy estimators, validating the bottleneck theory, potential training innovations. Further, some of your experiments aim to validate the replica formula where it is not known to hold,  e.g. during training (thanks to dependence). It would be good to add an experiments summary that clarifies these aims and points the readers to the relevant sections. You might have to move some of the  discussion due to space constraint.    I have read the author response. My scores are unchanged, I look forward to the final version of the article.