Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. We explain why this quantity makes our model able to achieve invariance with respect to input variations. We empirically validate the efficiency of our approach to improve classification performances compared to standard regularization schemes on several standard architectures.
Summary:
The paper presents an information theoretic regularizer for deep learning
algorithms. The regularizer aims to enforce compression of the learned
representation while conditioning upon the class label so preventing the
learned code from being constant across classes. The presentation of the Z
latent variable used to simplify the calculation of the entropy H(Y|C) is 
confusing and needs revision, but otherwise the paper is interesting.
Major Comments:
- The statement that I(X;Y) = I(C;Y) + H(Y|C) relies upon several properties
  of Y which are not apparent in the text (namely that Y is a function of X,
so I(X;Y) should be maximal, and Y is a smaller code space than X so it should 
be H(Y)). If Y is a larger code space than X then it should still be true, but
the logic is more complicated.
- The latent code for Z is unclear. Given the use of ReLUs it seems like Y
  will be O or +ve, and Z will be 0 when Y is 0 and 1 otherwise, so I'm
unclear as to when the value H(Y|Z) will be non-zero. The data is then
partitioned within a batch based on this Z value, and monte carlo sampling is
used to estimate the variance of Y conditioned on Z, but it's really unclear
as to how this behaves as a regularizer, how the z is sampled for each monte
carlo run, and how this influences the gradient. The discussion in Appendix C
doesn't mention how the Z values are generated.
- The discussion on how this method differs from the information bottleneck is
  odd, as the bottleneck is usually minimising the encoding mutual information
I(X;Y) minus the decoding mutual information I(Y;C). So directly minimising
H(Y|C) is similar to the IB, and also minimising H(Y|C) will affect I(C;Y) as
I(C;Y) = H(Y) - H(Y|C).
- The fine tuning experiments (Section 4.2) contain no details on the
  parameters of that tuning (e.g. gradient optimiser, number of epochs,
batch size, learning rates etc).
- Section 4.4 is obvious, and I'd consider it a bug if regularising with label
  information performed worse than regularising without label information.
Essentially it's still adding supervision after you've removed the
classification loss, so it's natural that it would perform better. This
experiment could be moved to the appendix without hurting the paper.
- In appendix A an upper bound is given for the reconstruction error in terms
  of the conditional entropy. This bound should be related to one of the many
upper bounds (e.g. Hellman & Raviv) for the Bayes rate of a predictor, as
there is a fairly wide literature in this area.
Minor Comments:
- The authors do not state what kind of input variations they are trying to
  make the model invariant to, and as it applies to CNNs there are multiple
different kinds, many of which are not amenable to a regularization based
system for inducing invariance.
- The authors should remind the reader once that I(X;Y) = H(Y) - H(Y|X) = H(X) -
  H(X|Y), as this fact is used multiple times throughout the paper, and it may
not necessarily be known by readers in the deep learning community.
- Computing H(Y|C) does not necessarily require computing c separate
  entropies, there are multiple different approaches for computing this
entropy.
- The exposition in section 3 could be improved by saying that H(X|Y) measures
  how much the representation compresses the input, with high values meaning
large amounts of compression, as much of X is thrown away when generating Y.
- The figures are difficult to read when printed in grayscale, the graphs
  should be made more readable when printed this way (e.g. different symbols,
dashed lines etc).
- There are several typos (e.g. pg 5 "staking" -> "stacking").