CAVEATS FOR INFORMATION BOTTLENECK IN DETERMINISTIC SCENARIOS
Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y . To do so, IB identifies an intermediate "bottleneck" variable T that has low mutual information I(X; T ) and high mutual information I(Y ; T ). The IB curve characterizes the set of bottleneck variables that achieve maximal I(Y ; T ) for a given I(X; T ), and is typically explored by maximizing the IB Lagrangian, I(Y ; T ) − βI(X; T ). In some cases, Y is a deterministic function of X, including many classification problems in supervised learning where the output class Y is a deterministic function of the input X. We demonstrate three caveats when using IB in any situation where Y is a deterministic function of X: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of β; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. We also show that when Y is a small perturbation away from being a deterministic function of X, these three caveats arise in an approximate way. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the three caveats on the MNIST dataset.
This paper is about issues that arise when applying Information Bottleneck (IB) concepts to machine learning, more precisely in deterministic supervised learning such as classification (deterministic in the sense that the target function to estimate is deterministic: it associates each example to one true label only, and not to a distribution over labels).
Namely:
(1) the "Information Bottleneck curve" cannot be computed with the Information Bottleneck Lagrangian approach (because of optimization landscape issues: optimization of such a piecewise-linear function with a linear penalty will always yield the same optimum whatever the slope of the penalty is [same story as L1 vs. L0]); 
(2) there are many solutions to the optimization of the IB Lagrangian for any given compression/performance ratio (i.e. for any given beta in the IB Lagrangian method: I(Y,T)/I(X,T)) and some of them are provably trivial; thus optimizing just the IB Lagrangian does not imply that the solution will be interesting, and better (or complementary) criteria are needed.
Another point discussed also is about the successive layers of perfect classifiers (neural networks), in which I(Y,T) remains constant while I(X,T) decreases.
Pros:
- the paper is well written, mostly self-contained, and easy to read (for someone familiar with information theory);
- all mathematical points are detailed and well explained, with sufficient introduction;
- the writing is compact, the paper is dense, and given the page limit this is a good information/compression compromise;)
- information bottleneck is a topic of prime interest in the community these days;
- the two first problems described ((1) and (2)) are original, interesting contributions to the field, of particular interest for people interested in applying information bottleneck concepts to supervised learning;
- the solution brought to the IB Lagrangian issues is simplistic though efficient (squaring I(X,T) so that it's not linear in I(X,T) anymore).
Cons:
- not much.
Remarks:
- there exist recent papers tackling the information bottleneck concept for neural networks from a variational perspective, which enables them to compute exactly the mutual informations (such as "Compressing Neural Networks using the Variational Information Bottleneck" by Dai & al., ICML 2018); I have not seen these papers cited in the article, nor discussed (nor used); I feel it would be appropriate, either in the general literature section, either for discussing how to compute in practice the mutual informations (exact values vs. estimates or lower bounds as here).
- at first reading, I had found the tone of the beginning of the paper (first section) a bit aggressive, though this feeling disappeared later. Maybe rephrase some expressions that might be wrongly perceived?
- About multilabel classification (end of section 2): multilabel classification can still be seen as with deterministic expected outputs, if considered as a task from X to P(Y) (power set of Y, i.e. set of all possible subsets of labels).
- As in practice T is constrained to belong to a particular space of functions (neural network layer with predefined architecture): how does this impact the study? For instance the T_alpha in equation (5) are not reachable anymore; the optimization space for the IB Lagrangian is different; etc. Which properties/conclusions can be kept, and which ones cannot?
- What about sampling on the other part of the IB curve, the horizontal one (same I(Y,T) for various I(X,T))? Would it bring any insight, and how to do it?
- A side remark about applying IB to neural networks: What about neural networks that are not a "linear" chain of layers (i.e. most networks now)? i.e. Inception, ResNet, U-nets, etc., where computational flows are parallel, sometimes keeping full information till the end. For instance in a U-net, meant for image processing, features computed at the beginning at a full pixelic resolution are communicated to the last layer. This is not an image classification task though, as predictions are made for each pixel; still, given an input image X, there is only one correct output Y, so, still in the deterministic supervised classification problem.