PARAMETRIC INFORMATION BOTTLENECK TO OPTIMIZE STOCHASTIC NEURAL NETWORKS
In this paper, we present a layer-wise learning of stochastic neural networks (SNNs) in an information-theoretic perspective. In each layer of an SNN, the compression and the relevance are defined to quantify the amount of information that the layer contains about the input space and the target space, respectively. We jointly optimize the compression and the relevance of all parameters in an SNN to better exploit the neural network's representation. Previously, the Information Bottleneck (IB) framework (Tishby et al. (1999)) extracts relevant information for a target variable. Here, we propose Parametric Information Bottleneck (PIB) for a neural network by utilizing (only) its model parameters explicitly to approximate the compression and the relevance. We show that, the PIB framework can be considered as an extension of the maximum likelihood estimate (MLE) principle to every layer level. We also show that, as compared to the MLE principle, PIB : (i) improves the generalization of neural networks in classification tasks, (ii) is more efficient to exploit a neural network's representation by pushing it closer to the optimal information-theoretical representation in a faster manner.
This paper proposes a learning method (PIB) based on the information bottleneck framework.
PIB pursues the very natural intuition outlined in the information bottleneck literature: hidden layers of deep nets compress the input X while maintaining sufficient information to predict the output Y.
It should be noted that the limitations of the IB for deep learning are currently under heavy discussion on OpenReview.
Optimizing the PIB objective is intractable and the authors propose an approximation that applies to binary valued stochastic networks.
They use a variational bound to deal with the relevance term, I(Z_l,Y), and  Monte Carlo sampling to deal with the layer-by-layer compression term, I(Z_l,Z_{l+1}).
They present results on MNIST aiming to demonstrate that using PIBs improves generalization and training speed.
This is a timely and interesting topic. I enjoyed learning about the authors’ proposed approach to a practical learning method based on the information bottleneck. However, the writing made it challenging and the experimental protocol raised some serious questions. In summary, I think the paper needs very careful editing for grammar and language and, more importantly, it needs solid experiments before it’s ready for publication. When that is done it would make an exciting contribution to the community. More details follow.
Comments:
1. All architectures and objectives (both classic and PIB-based) are trained using a single, fixed learning rate (LR). In my opinion, this is a red flag. The PIB objective is new and different to the other objectives. Do all objectives happen to yield their best performance under the same LR? Maybe so, but we won’t know unless the experimental protocol prescribes a sufficient range of LRs for each architecture. In light of this, the fact that SFNN is given extra epochs in Figure 4 does not mean much.
2. The batch size for MNIST classification is unusually low (8). Common batch sizes range from 64 to 1K (typically >= 128). Why did the authors make this choice? Is 8 good for architectures A through E?
3. On a related note, the authors only seem to report results from a single random seed (ie. deterministic architectures are trained exactly once). I would like to see results from a few different random seeds. As a result of comments 1,2,3, even though I do believe in the merit of the intuition pursued and the techniques proposed, I am not convinced about the main claim of the paper. In particular, the experiments are not rigorous enough to give serious evidence that PIBs improve generalization and training speed. 
4. The paper needs some careful editing both for language (cf. following point) but also notation. The authors use notation p_D() in eqn (12) without defining it. My best guess is that it is the same as p_u(), the underlying data distribution, but makes parsing the paper hard. Finally there are a few steps that are not explained: for example, no justification is given for the inequality in eqn (13).
5. Language: the paper needs some careful editing to correct numerous language/grammar issues. At times it is detrimental to understanding. For example I had to read the text leading up to eqn (8) a number of times.
6. There is no discussion of computational complexity and wall-clock time comparisons. To be clear, I think that even if the proposed approach were to be slower than the state of the art it would still be very interesting. However, there should be some discussion and reporting of that aspect as well.
Minor comments and questions:
7. Mutual information is typically typeset using a semicolon instead of a comma, eg. I(X;Z).
8. Why is the mutual information in Figure 3 so low? Are you perhaps using natural logarithms to estimate and plot I(Z;Y)? If this is base-2 logarithms I would expect a value close to 1.