We formulate an information-based optimization problem for supervised classification. For invertible neural networks, the control of these information terms is passed down to the latent features and parameter matrix in the last fully connected layer, given that mutual information is invariant under invertible map. We propose an objective function and prove that it solves the optimization problem. Our framework allows us to learn latent features in a more interpretable form while improving the classification performance. We perform extensive quantitative and qualitative experiments in comparison with the existing state-of-the-art classification models.
In this paper, the authors propose to train a model from the point of view of maximizing mutual information between the predictions and the true outputs, with a regularization term that minimizes irrelevant information while learning. They show that the objective can be minimized by looking to make the final layer vectors be as uncorrelated as possible to the final layer representations, and simplify the same by applying Holder’s inequality to make the optimization tractable. They also apply an L1 penalty on the final layer. Experiments on CIFAR and MNIST show that using their regularizer to train DNN models yield  gains in performance. The presence of the L1 penalty also makes the results more interpretable (to the extend possible by looking at a subset of features in the last layer of a DNN). 
COMMENTS:
- Meta Point: To really see that the regularization framework you’re proposing is good, why not just pick a simple, feedforward model or convNet, see the performance and then compare it with the regularizer you’re proposing? That will help hit the point home. 
- Page 1: before jumping to equations (1) and (2), please formally define Mutual Information. The actual definition is much later in the text, but it’s better to define first. 
- Beyond  referring the user to section 3 on Page 1, please also mention a couple of key references in the appropriate locations. 
- Page 3 paragraph 2: “Mutual information is bounded … correct them” : Can you provide some formulas for this and make this concrete? Or perhaps provide some references? This line is vague. 
 - prop 2.1 and 2.2: can you define what you mean by “empirical version”? Again, it’s probably good to have these terms crisply defined before using them. 
- eqn (6) is interesting. Holder’s inequality gives you the product terms. Then you can also apply the AM-GM inequality, and get a sum. So then at the end of it all, you’re left with the standard elastic net penalty and not the product form. In that case, aren’t we back to just the usual regularization strategy? And in which case, should I interpret the results you have in sec 4 as “using L1 penalties with L2 is good” ?
- To the point above, I guess one difference after the AM-GM step is that you will not have a squared L2 norm, but just L2. This is reminiscent of linear models where they use L2 loss instead of squared L2 loss. But on the penalty, squaring just adds smoothness. Can you comment on this? 
- sec 4.2.1: I don’t see how fig (2) (L) is “roughly Gaussian”. Can you explain? Maybe plot the histogram? Also for fig (2, R): the coefficients are approximately sparse. It’s not sparse as you claim since there are almost no zeros in the coefficients. 
- I don’t get the point of sec 4.3: How does this claim not apply to all deep learning models, regardless of the penalizations you propose?
edit: 
I have read the responses and the other reviews. The authors have addressed the few major points I had. I still think there are a few gaps that need to be addressed (as pointed by the other reviewers)