It has been shown that instead of learning actual object features, deep networks tend to exploit non-robust (spurious) discriminative features that are shared between training and test sets. Therefore, while they achieve state of the art performance on such test sets, they achieve poor generalization on out of distribution (OOD) samples where the IID (independent, identical distribution) assumption breaks and the distribution of non-robust features shifts. Through theoretical and empirical analysis, we show that this happens because maximum likelihood training (without appropriate regularization) leads the model to depend on all the correlations (including spurious ones) present between inputs and targets in the dataset. We then show evidence that the information bottleneck (IB) principle can address this problem. To do so, we propose a regularization approach based on IB called Entropy Penalty, that reduces the model's dependence on spurious featuresfeatures corresponding to such spurious correlations. This allows deep networks trained with Entropy Penalty to generalize well even under distribution shift of spurious features. As a controlled test-bed for evaluating our claim, we train deep networks with Entropy Penalty on a colored MNIST (C-MNIST) dataset and show that it is able to generalize well on vanilla MNIST, MNIST-M and SVHN datasets in addition to an OOD version of C-MNIST itself. The baseline regularization methods we compare against fail to generalize on this test-bed.
The paper proposes to combat domain shift via the information bottleneck principle, where the representations are encouraged to learn features that have high mutual information with the labels while having low mutual information with the original data. The paper proposes a specific approach to enforcing this information bottleneck, called "entropy penalty", which tries to minimize the L2 distance between the representation at the first layer and the mean representation of the corresponding class. 
The paper proceeds to demonstrate theoretically that some variant of the proposed entropy regularization works for synthetic, binary cases in the sense that it decreases the effect of features that are not highly correlated with the label.
The paper finally demonstrates superior performance on synthetic experiments and classifying digits in an out-of-distribution setting where the OOD distribution is changing foreground and background color.
My greatest concern of the proposed approach is the generality of the method. While the information bottleneck is a very general principle, the way it is implemented in this paper is very specific, which only works for the first hidden layer representation of an input x (even before non-linearity). Combining this with the experiment in which the domain shift is changing colors of in the images (which applies linear transforms to background and foreground pixels), it is hard for me to believe that the same procedure is not tailored for this specific type of domain shift (although this is still an interesting type of domain shift that is important in sim-to-real applications in robotics). 
In fact, based on the information bottleneck principle, one could essentially learn that the color is also a feature that also has high MI with label and low MI with inputs, thus the top-left pixel would be indicative of the label. If I use this as the feature for entropy penalty (assuming that there is 1 color per class), this gives a R_{EP} loss of zero. I wonder if this is the reason why the experiments specifically asked for two foreground and two background colors for each class.
Based on there concerns, it might seem more helpful to consider other types of "unknown" domain shifts, such as CIFAR vs. CINIC, in which the method proposed might introduce fewer inductive biases than in the case of being invariant to color. 
Minor comments:
	- It would seem like L1 regularization would achieve a similar effect to what is shown in the theory here? It would be interesting to see how sensitivity changes with some existing regularization methods, because they can be easily implemented.
	- Entropy is smaller for lower layers?
	- Minimizing entropy for higher layers at least minimizes an upper bound to the entropy of lower layers (if we consider discrete RVs).
	- Eq 6 is false for continuous RV and differential entropy. One could simply have h_2 = 2 * h_1 to get larger entropy. Eq 5 is true from data processing but the conditional entropy H(h_l | x) is not fixed.
	- It seems that from the discussion about entropy / MI of different layers does not justify the EP approach for the first layer. Even if the final layer retains all the information, you can still technically apply EP to remove redundant information? You can use leaky relu activation to make sure information is not lost in activation functions.
	- Typo for definition of \mu_k
	- The first layer is a convolution layer, so maybe it could be helpful to visualize the learned features with or without EP?