The recently proposed information bottleneck (IB) theory of deep nets suggests that during training, each layer attempts to maximize its mutual information (MI) with the target labels (so as to allow good prediction accuracy), while minimizing its MI with the input (leading to effective compression and thus good generalization). To date, evidence of this phenomenon has been indirect and aroused controversy due to theoretical and practical complications. In particular, it has been pointed out that the MI with the input is theoretically infinite in many cases of interest, and that the MI with the target is fundamentally difficult to estimate in high dimensions. As a consequence, the validity of this theory has been questioned. In this paper, we overcome these obstacles by two means. First, as previously suggested, we replace the MI with the input by a noise-regularized version, which ensures it is finite. As we show, this modified penalty in fact acts as a form of weight-decay regularization. Second, to obtain accurate (noise regularized) MI estimates between an intermediate representation and the input, we incorporate the strong prior-knowledge we have about their relation, into the recently proposed MI estimator of Belghazi et al. (2018). With this scheme, we are able to stably train each layer independently to explicitly optimize the IB functional. Surprisingly, this leads to enhanced prediction accuracy, thus directly validating the IB theory of deep nets for the first time.
While overall the writing quality of the paper is high, the paper itself is a strong rejection.  I believe the analysis of the paper is at points flawed, and the experiments are minimal.  
This work attempts to study the degree to which a layer by layer information bottleneck inspired objective can improve performance, as well as generally attempt to clarify some of the discussion surrounding Shwartz-Ziv & Tishby 2017.  Here, the authors study a deterministic neural network, for which the mutual information estimation is difficult (I(X,L)) and error prone.  To combat this they use the noise-regularized mutual information estimator (I(X; L+eps)).  To actually estimate the mutual information the authors use the MINE estimator of Belghazi (2018).  Here they suggest using the neural network itself as a structural element in the form of the discriminator to take advantage of the specific circumstances in this case.  Doing this ensured that their estimator diverged in the zero noise limit as expected.  From here they show some experimental results of the effect of their objective on an MNIST / CIFAR10 classification task.
This paper fits into what is an increasingly large discussion in the literature, surrounding Information Bottleneck.  The paper itself does a very good job of citing recent relevant work.  Technically however I take issue with the framing of previous work in the last paragraph of the "Deep neural nets" subsection of Section 2.  Technically Achille & Soatto explicitly formed a variational approximation to the posterior over the weights of the neural network and so was not a "single bottleneck layer" as stated in the paper.  More generally at the end of that paragraph it is implied that the single bottleneck layer scheme "deviates from the original theory".  This is a misleading characterization of the original information bottleneck (Tishby et al 1999) in which there was a single random variable, a representation of the data (Z) satisfying the Markov conditions Z <- X -> Y.   I believe the authors instead meant to say that the cited works deviate from the information bottleneck theory of learning suggested in (Shwartz-Ziv & Tishby 2017).  In general the paper does a poor job of distinguishing between the Shwartz-Ziv & Tishby paper and the rest, but this is a distinction that should be maintained.  The original information bottleneck may and has demonstrated utility regardless of whether the information bottleneck generally can help explain why ordinary deterministic feed forward networks trained with cross entropy and sgd generalize well.  
This also raises one of the main problems with the current work. The title, abstract and especially the conclusion ("This provides, for the first time, strong and direct emperical evidence for the validity of the IB theory of deep learning") seem to present the paper as somehow offering some clarity and further support for the assertions of the Shwartz-Ziv & Tishby 2017 paper, but that paper hoped to establish that information bottleneck can explain the workings of ordinary networks.  Here the authors modify the ordinary cross entropy objective, and so their networks are necessarily not ordinary and so they cannot claim they have helped clarify our understanding of the vast majority of neural networks currently being trained.  Again, this is distinct and should be kept distinct from the utility of their proposed objective, itself inspired by the information bottleneck.  Here too the paper falls flat.  If instead of attempting to comment on networks as they are designed today they aim to proposed a new information bottleneck inspired objective they really ought to directly compare other attempts along those lines (such as the ones they themselves cite  Alemi et al. 2018, Kolchinsky et al. 2017, Chalk et al. 2016, Achile & Soatto 2018, Belghazi et al. 2018) but there are no comparative studies.
The experiments are extremely lacking, not only are any of their cited alternatives compared, they don't compare to what would be an equivalent network to their but where they did utilize the noise at every layer and actually made the network stochastic.  Their reported numbers are not very impressive with their top MNIST number at 98.09 and their baseline at 97.73. These numbers are worse than many of the papers they themselves cite.  Only a single comparative results for both a limited training set run and the full one are shown, as well as only a single choice of beta.  The CIFAR10 numbers are not very good either.  There is some discussion of the text suggesting they believe their method acts like an approximate weight decay, but there are no results showing the effect of weight decay just on the baseline classification accuracies they compare against.
Technically a deterministic function need not have infinite mutual information, if it is non-invertible, i.e. the sign function, or just floating point discretization. 
Their own results in Figure 2 and the main body of the text highlight that the authors believe the true mutual information between the activations of the intermediate layers and the input is infinite.  If the true mutual information is infinite and the noise regularized estimator is only meant for comparative purposes, why then are the results of the training trajectories interpreted so literally as estimates of the true mutual information?
Just plugging in the Discriminator for the objective (equation (7)) is flawed.  The discriminator, if optimal would learn to approximate the density ratio 1 + log p(x,y)/(p(x) p(y)) .   ( see f-GAN, Norowin et al. 2016).  How does this justify using the individual elements of the discriminator in the functional form of the IB objective?  
At the bottom of page 6 they rightfully say that mutual information is invariant to reparameterizations, but their noise regularized mutual information estimator is not (by their own reference (Saxe et al 2017).
The discussion at the center of page 8 is confusing.   They claim that Figure 5 (a) is more 'quantized' than (b) and "has reduced entropy".  I think it should be the other way.  More clusters should translate to a higher KL divergence, or higher entropy.  If you need only identify which cluster an activation is in, that should require log K nats where K is the number of clusters.  (a) shows more clusters and so seems like it should cost more and have a higher entropy not a lower one.
Despite a recurring focus of the text that this paper applies and information theoretic objective at each layer of the network, and hence is novel, the final sentence of the paper suggests it might not actually be needed and single layer IB objectives can work as well.