We present a new family of objective functions, which we term the Conditional Entropy Bottleneck (CEB). These objectives are motivated by the Minimum Necessary Information (MNI) criterion. We demonstrate the application of CEB to classification tasks. We show that CEB gives: well-calibrated predictions; strong detection of challenging out-of-distribution examples and powerful whitebox adversarial examples; and substantial robustness to those adversaries. Finally, we report that CEB fails to learn from information-free datasets, providing a possible resolution to the problem of generalization observed in Zhang et al. (2016).
[UPDATE]
I find the revised version of the paper much clearer and streamlined than the originally submitted one, and am mostly content with the authors reply to my comments. However, I still think the the work would highly benefit from a non-heuristic justification of its approach and some theoretic guarantees on the performance of the proposed framework (especially, in which regimes it is beneficial and when it is not). Also, I still find the presentation of experimental results too convoluted to give a clear and comprehensive picture of how this methods compares to the competition, when is it better, when is it worse, do the observations/claim generalize to other task, and which are the right competing methods to be considering. I think the paper can still be improved on this aspect as well. 
As I find the idea (once it was clarified) generally interesting, I will raise my score to 6.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The paper proposes an objective function for learning representations, termed the conditional entropy bottleneck (CEB). Variational bounds on the objective function are derived and used to train classifiers according to the CEB and compare the results to those attained by competing methods. Robustness and adversarial examples detection of CEB are emphasized.
My major comments are as follows:
1) The authors base their 'information-theoretic' reasoning on the set-theoretic structure of Shannon’s information measures. It is noteworthy that when dealing with more than 2 random variables, e.g., when going from the twofold I(X;Y) to the threefold I(X;Y;Z), this theory has major issues. In particular, there are simple (and natural) examples for which I(X;Y;Z) is negative. The paper presents an information-theoretic heuristic/intuitive explanation for their CEB construction based on this framework. No proofs backing up any of the claims of performance/robustness in the paper are given. Unfortunately, with such counter-intuitive issues of the underlying theory, a heurisitc explanation that motivates the proposed construction is not convincing. Simulations are presented to justify the construction but whether the claimed properties hold for a wide variety of setups remain unclear.
2) Appendix A is referred to early on for explaining the minimal necessary information (MNI), but it is very unclear. What is the claim of this Appendix? Is there a claim? It's just seems like a convoluted and long explanation of mutual information. Even more so, this explanation is inaccurate. For instance, the authors refer to the mutual information as a 'minimal sufficient statistic' but it is not. For a pair of random variables (X,Y), a sufficient statistic, say, for X given Y is a function f of Y such X-f(Y)-Y forms a Markov chain. Specifically, f(Y) is another random variable. The mutual information I(X;Y) is just a number. I have multiple guesses on what the authors' meaning could be here, but was unable to figure it out from the text. One option, which is a pretty standard way to define sufficient statistic though mutual information is as a function f such that I(X;Y|f(Y))=0. Such an f is a sufficient statistic since the zero mutual information term is equivalent to the Markov chain X-f(Y)-Y from before. Is that what the authors mean..?
3) The Z_X variable introduced in Section 3 in inspired by the IB framework (footnote 2). If I understand correctly, this means that in many applications, Z_X is specified by a classifier of X wrt the label Y. My question is whether for a fixed set of system parameters, Z_X is a deterministic function of X? If this Z_X play the role of the sufficient statistics I've referred to in my previous comment, then it should be just a function of X. 
However, if Z_X=f(X) for a deterministic function f, then the CEB from Equation (3) is vacuous for many interesting cases of (X,Y). For instance, if X is a continuous random variable and Z_X=f(X) is continuous as well, then 
I(X;Z_X|Y)=h(Z_X|Y)-h(Z_X|X,Y)
where h is the differential entropy and the subtracted terms equals -\infty by definition (see Section 8.3 of (Cover & Thomas, 2006). Consequently, the mutual information and the CEB objective are infinite. If Z_X=f(X) is a mixed random variable (e.g., can be obtain from a ReLU neural network), then the same happens. Other cases of interest, such as discrete X and f being an injective mapping of the set of X values, are also problematic. For details of such problem associated with IB type terms see:
[1] R. A. Amjad and B. C. Geiger 'Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle', 2018 (https://arxiv.org/abs/1802.09766).
Can the authors account for that?
4) The other two reviews addressed the missing accounts for past literature. I agree on this point and will keep track of the authors' responses. I will not comment on that again. 
Beyond these specific issue, they text is very wordy and confusing at times. If some mathematical justification/modeling was employed the proposed framework might have been easier to accept. The long heuristic explanations employed at the moment do not suffice for this reviewer. Unless the authors are able to provide clarification of all the above points and properly place their work in relation to past literature I cannot recommend acceptance.