The information bottleneck principle (Shwartz-Ziv & Tishby, 2017)  suggests that SGD-based training of deep neural networks results in optimally compressed hidden layers, from an information theoretic perspective. However, this claim was established on toy data. The goal of the work we present here is to test whether the information bottleneck principle is applicable to a realistic setting using a larger and deeper convolutional architecture, a ResNet model. We trained PixelCNN++ models as inverse representation decoders to measure the mutual information between hidden layers of a ResNet and input image data, when trained for (1) classification and (2) autoencoding. We find that two stages of learning happen for both training regimes, and that compression does occur, even for an autoencoder. Sampling images by conditioning on hidden layers' activations offers an intuitive visualisation to understand what a ResNets learns to forget.
Conditionally generated images are shown in (b) -(f). Ten epochs is the peak of fitting, and 200 epochs is the end of compression. These samples enable an intuitive illustration of compression in hidden layers. Based on this example it seems that a compressed representation (f) results in varied samples because it compresses class-irrelevant information. Compare the beginning (d) to the end (f) of compression: there is greater variety at the end without losing the essence of 'horse'.
Deep neural networks are ubiquitous in machine learning and computer vision. Unfortunately, the popularity of neural networks for applications is unmatched by an agreed upon and clear understanding of how exactly they work, and why they generalise well. The field of deep learning will advance with more comprehensive theory and empirical studies that better characterise neural networks.
Generalisation, in the context of learning, means the extraction of typical abstract properties of training data that can be successfully used for inference on unseen data. Neural networks generalise well even when they are over-parameterised. Zhang et al. (2017) highlighted the need to rethink generalisation, because conventional wisdom is not readily applicable to neural networks. A number of avenues for new generalisation bounds for neural networks (Bartlett et al., 2017;Golowich et al., 2017;Neyshabur et al., 2017) exemplify how inapplicable conventional methods for understanding model generalisation can be.
One approach to better understanding neural networks is the Information Bottleneck (IB, Section 2) interpretation of deep learning (Tishby et al., 2000;Tishby & Zaslavsky, 2015;Shwartz-Ziv & Tishby, 2017). The IB accredits deep learning success to compression in hidden layers via the noise of parameter updates in stochastic gradient descent (SGD). Information compression results in optimal representations that discard task-irrelevant data while keeping task-relevant data.
The IB principle has since been actively debated (Saxe et al., 2018), partially motivating this work.
The novelty is that we apply information theoretic analyses to modern convolutional residual neural networks (ResNets, Section 5) trained on realistic images (Section 5.2). These choices complicate analysis since information quantities are non-trivial to compute for high-dimensions. Our solution is to define a lower bound on the mutual information (MI, Section 4) and to estimate it by training decoder models (Section 4.1). The decoder model for the MI between images and hidden layers is a conditional PixelCNN++ and samples generated therefrom illustrate visually MI ( Figure  1).
* Summary: 
This work is an empirical study of the relevance of the Information Bottleneck principle as a way of understanding deep-learning. It is carried out in the setting of realistically sized networks trained on natural images dataset. This is, in spirit, a meaningful and sensible contribution to the ongoing debate. Being a largely empirical contribution, its value hinges on the exhaustivity and clarity of the experiments carried out. As it stands, I believe that these should be, and can be, improved. Details to support this opinion are given below. A summary of my expectations is given at the end.
* Summary of the approach and significance:
The IB principle relies on estimating the mutual information between i) the input and an intermediate layer, I(x,h), and an intermediate layer and the output, I(h, y). Previous work has relied on binning strategies to estimate these quantities. This is not applicable in a real-sized problem such as classification of natural images with deep networks. This paper proposes to invert a first deep model using a second, generative, model which must reconstruct the input of the first given some intermediate layer. The information progressively discarded by the first network should be modelled as uncertainty by the second. This yields a lower bound on the mutual information, with a tightness that depends on the expressivity of the generative model.
I believe the goal to be meaningful and a valuable contribution: going forward, testing this assumption in realistic setting is essential to the debate. The proposed approach to do this seems sensible to me. It is similar to cited work by Nash et al., however both works are concurrent and so far unpublished and should be considered as complementary point of views on the same problem.
Partial conclusion: The goal is meaningful and sensible.
* Quality of writing.
In sections 1, 2 and 3, the motivation is clear and contains relevant information. I feel it could be polished further. In particular, some redundant information could be condensed. The introduction to the IB principle, though understandable, could be improved. Here are some opinions:
>> Paragraphs 3 and 4 of 1.0: A reference to M. Saxe et al. could already be made there: it is a major 'opponent' in the debate to which you are empirically contributing.
>> In paragraph 1.1: Points 1) and 2) are redundant. So are 3) and 4). Paragraph 1.1 as a whole is largely redundant with the previous paragraph, these could be collapsed. 
>> In section 2: I feel that the main intuitions of the encoder / decoder distributions (I(y,h) / I(x,h)), the information plane, and the optimal bottleneck representations (Sections 2.1 to 2.3 of Schwartz-Ziv & Tishby) could be better conveyed in 2, though I understand the need for brevity. 
>> Related works: Descriptions of related works could be condensed. On the other hands, the points of these papers that you contradict in you experiment section could be explicitly mentioned again there.
>> End of section 3, section 4: The fact that estimation of the MI by traditional means is not applicable in your setting is repeated many times throughout the paper, in noticeably rapid succession in that region. Some mentions should be removed.
	
### More minor points:
>> Paragraphs 2 of 1.0:
- 'the extraction of typical abstract properties...' this statement is vague and thus debatable: the channel-wise mean in RGB space, for instance, is not an especially abstract property. 
- Reference is made to Zhang et al. to justify the need for more analysis of generalization in deep-learning. This paper can be considered controversial. Mention could be made of other works, for instance about the inaplicability of traditional generalization bounds and attempts to improve them.
>> Pseudo code algorithm.
- Could be summarized in the main body and pushed to the annex.
>> The choice of pixCNN as a generative model could be discussed more. There are some good reasons to prefer this to L2 regression for instance. 
Partial conclusion: The description of the method contains relevant information and is functional, but the writing could be improved.
	
	
* Experimental results.
> The contribution and novelty of this paper is largely empirical. Therefore the experimental results should be held to a high standard of clarity and exhaustivity.
*** The choice of dataset:
The experimental setup seems to be fair in terms of dataset / split chosen: the abundance of data for the three steps (encoding, decoding, evaluation) is a notable strength.
*** The quality of the lower bound: Uncertainty when reconstructing the image may come from the fact that information has been discarded. Variance may also come from the pixCNN++, which is imperfect. You mention this (paragraph 4.1) but do not take experimental steps to measure it. Please consider reporting the performance of your generative model i) without conditioning, ii) conditioned on one-hot ground truth labels, and optionally iii) on grayscale/downsampled versions of the image without otherwise modifying the training setup. These values will give the reader an idea of the significance of variations in MI measured and give a 'scale' to your figures, strengthening your claims.
*** The evolution of compression *accross iterations* for a fixed layer
I will focus on the classification setting for now.
Qualitatively: Figures 1, 4 and 5 do not convince me that a meaningful evolution in the type of information discarded *across training iterations* can be observed visually. In figures 1 and 4, the network seems to learn invariances to some coloring, and its preferred colours vary across iterations. Beyond that I cannot see much, except maybe for column (f) of figure 4, despite your claim in section 2, paragraph 2.
Quantitatively: Curves in Figure 2 a) are more convincing, though a notion of scale is missing, as already discussed. The evolution of I(y; h) across iterations is very clear, in Figure 2 a) and especially 3 a). The evolution of I(x, h) much less so. h3 and h4 do not seem to show anything meaningful. In h2 the decrease in I(x, h) is supported by only 2 points in the curve (epoch 10 to epoch 100, and epoch 100 to epoch 200, figures 2a and 3c). Epochs displayed are also incoherent from one curve to the next (epoch 15 is missing for h2 in fig 3c) which raises suspicion. It appears important to i) display more points, to show that this is not just noise and ii) track more layers to confirm the trend, supported by a single layer so far (see next paragraph). I understand that each of these points require training of a generative model, but I feel it is necessary to make reliable conclusions.  
Minor: In figure 2, epochs should be added as labels to the colours.  
*** The evolution of compression *across layers* for a fixed iteration
		
Conversely, the evolution of the MI across layers is very convincingly demonstrated, and I feel this is perhaps the main strength of this paper. All curves display consistent trends across layers, and Figure 5 qualitatively displays much more invariance to pose, detail, etc than Figure 4. This is interesting, and could be made more central: i) by making a figure that compares samples across layers, for a fixed iteration, side by side. 
On the downside, I believe it is important to track more layers, as it is to me the main interest of your results. The second paragraph of section 5 does not give a good idea of the spread of these layers to someone not familiar with the resnet architecture used. For example, the penultimate layer of the network could be used (the layer at which the most compression is to be expected).
*** On the auto-encoder experiments.
> Little detail is given about the way the auto-encoder is constructed. In particular, one expects the type of bottleneck used (necessary so that the network does not learn the identity function) to have large impact on the amount of information discarded in the encoding process. This dependency is not discussed. More crucially, experiments with different types / strength of bottleneck are not given, and would, in my opinion, be key to an analysis of this dependency through the IB principle. 
> Furthermore, no qualitative analysis is provided in this setting.
> Without these additions, I find the Auto-encoding setting an unconvincing distraction from the main contribution of this paper. 	
***  main avenues of improvement:
		
> Two kinds of progression in compression are demonstrated in your paper: across layers, and across iterations. 
As it stands, results evidence the former more convincingly than the latter, both qualitatively and quantitatively.
I believe results could be presented in a way that clearly takes better advantage of this, as I will detail further.
More data points (across layer and epochs) would be beneficial. I feel that the auto-encoder setting, as it stands, is a distraction.
I would find this paper more convincing if experiments focused more on showing how layers progressively discard information, and less on the 'training phases' that are so far less clear.
*** Additional comments
The following are a number of points that would be worthwhile to discuss in the paper
> As it stands, it seems the line of reasoning and experimental setup seems to rely on the chain-structured nature of the considered neural net architecture. Can the same line of reasoning be applied to networks with more general computational graphs, such as dense-nets [a], mulit-scale denseness [b], fractal nets [c] etc.
[a] Huang, G.; Liu, Z.; van der Maaten, L. & Weinberger, K. Densely connected convolutional networks CVPR, 2017
[b] Huang, G.; Chen, D.; Li, T.; Wu, F.; van der Maaten, L. & Weinberger, K. Multi-Scale Dense Networks for Resource Efficient Image Classification ICLR, 2018
[c] https://arxiv.org/abs/1605.07648
> Why is it that earlier layers are estimated to have larger MI with the target y than later layers before convergence? Sure, later layers compress certain information about the input x, which could be informative on the response variable y. But since the MI estimate for early layers depends on the same network architecture as the one used to compute the later layers from the early ones, the result seems counter intuitive. See paragraph "forward direction" in section 4.1.
> The orange curve in fig 3a estimating I(x;y) is not commented upon. How was it obtained, what is its relevance to the discussion?