It has been common to argue or imply that a regularizer can be used to alter a statistical property of a hidden layer's representation and thus improve generalization or performance of deep networks. For instance, dropout has been known to improve performance by reducing co-adaptation, and representational sparsity has been argued as a good characteristic because many data-generation processes have only a small number of factors that are independent. In this work, we analytically and empirically investigate the popular characteristics of learned representations, including correlation, sparsity, dead unit, rank, and mutual information, and disprove many of the conventional wisdom. We first show that infinitely many Identical Output Networks (IONs) can be constructed for any deep network with a linear layer, where any invertible affine transformation can be applied to alter the layer's representation characteristics. The existence of ION proves that the correlation characteristics of representation can be either low or high for a well-performing network. Extensions to ReLU layers are provided, too. Then, we consider sparsity, dead unit, and rank to show that only loose relationships exist among the three characteristics. It is shown that a higher sparsity or additional dead units do not imply a better or worse performance when the rank of representation is fixed. We also develop a rank regularizer and show that neither representation sparsity nor lower rank is helpful for improving performance even when the data-generation process has only a small number of independent factors. Mutual information I(z l ; x) and I(z l ; y) are investigated as well, and we show that regularizers can affect I(z l ; x) and thus indirectly influence the performance. Finally, we explain how a rich set of regularizers can be used as a powerful tool for performance tuning.
This paper investigates the use of techniques for improving neural network training (regularization, normalization of covariance, sparsity) in terms of their generalization properties, empirically and analytically. The claim is that most of these tools do not help improve performance, with the exception of mutual information.
Pros: It's interesting to investigate and compare these different "regularization" techniques and compare them on different tasks empirically.
Cons:
Many of the points made in the paper are not properly capturing the nuance in the "conventional wisdom", and although it's good to be reminded and the empirical results are interesting to look at, in fact these are not really new discoveries, and sometimes the conclusions are very misleading. 
1) There is test loss, and there is generalization loss, and it isn't exactly the same thing. For a hypothesis class H, we have
test loss <= train loss + generalization loss 
where train loss measures how well we've fit a particular sample, and generalization loss measures how well a model that is trained on one sample can fit a new sample. Note that if I apply L2 loss to my model and have a regularization parameter --> infty, my train loss is huge but my generalization loss is 0. In other words, for a large enough regularization parameter, most of the methods experimented here WILL limit effective capacity and minimize generalization loss; it just will not give you the best test error performance. So the distinction here should be made much clearer--conventional wisdom for regularization limiting generalization error is not wrong.
2) The point that is trying to be made in section 3.1 is somewhat well-known in the general optimization literature. Let's consider a much simpler example: linear regression 
min_x ||Ax-b||_2^2 + gamma ||x||_2^2
Let's consider first no regularization, gamma = 0. Assume that A has a huge nullspace. Then technically there are an infinite number of globally optimal solutions x, although if we solve this problem using SGD starting with x = 0, it is known that the minimum norm solution is always picked. You can also think of this as whitening, since large lambda smooths the spectrum of the Hessian. Now add in regularization. Now the solution is unique, even if A is ill-conditioned. It's true that it isn't super necessary to add this regularization, since SGD can get you a good solution, but now we can GUARANTEE that the generalization error is 0. In practice, also, regularization adds stability to the numerics. 
In deep learning, the hidden layer z also acts as a coefficient matrix for determining y. I assume that is why people pursue low correlation, since it affects the conditioning of z. 
3) Comments like  "for scaling and permutation, their influences are rather insignificant" seem a bit careless to me. In fact it is well known that scaling can affect training performance significantly. But of course, if I know the global solution for z which feeds into a softmax, then any scaling on z does not affect the output of the softmax. That, however, does not mean I don't care about the scaling of z when training. 
4) Sparsity and rank BOTH limit the degrees of freedom. In fact, sparsity makes more sense when optimizing a nonlinear objective, which is always the case in deep learning. The reason to limit rank is when you wish to "learn your codewords", e.g. the eigenvectors, whereas in sparsity, the "codewords" are already learned, and you just learn the weighting. But if the codewords span the space, they both have equal representability. 
4) It is not clear to me what the task is in 4.3. What is the "accuracy" in a data generation task? Is this the normal classification task? If so, is the accuracy reported train or test accuracy? How exactly is generalization error being measured? 
5)I am not clear as to what conclusion is being drawn in section 5, with the last sentence "obviously, many of the statistical characteristics become meaningless for such a scalar representation, and it is high time to reconsider the so-called conventional wisdom on representation characteristics." why is this conclusion drawn based on the observation that, if a scalar z perfectly correlates with y, in fact this is the most generalization neural network? 
6) Table 4: how did you choose your hyperparameters? (regularization performance is extremely sensitive to parameter choice.)
7) A major concern is that basically very little training is done in these comparisons, except in the very last section. As I previously mentioned, many of these regularization / normalization techniques are also meant to better condition the optimization itself, and thus this advantage should not be discarded. 
minor comments:
 - page 5 last sentence "characteristics" should be singular
 - page 8 first sentence "to [a] deep network's performance"