We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. We study the prevalence of local minima in loss landscapes, whether small-norm parameter vectors generalize better (and whether this explains the advantages of weight decay), whether widenetwork theories (like the neural tangent kernel) describe the behaviors of classifiers, and whether the rank of weight matrices can be linked to generalization and robustness in real-world networks.
The authors look at empirical properties of deep neural networks and discuss their connection to past theoretical work on the following issues:
* Local minima: they give an example of setting where bad local minima (far from the global minimum) are obtained. More specifically, they show such minima can be obtained by initializing with large random biases for MLPs with ReLU activation. They also provide a theoretical result that can be used to find a small set of such minima. I believe this is a useful incremental step towards a better understanding of local minima in deep learning, although it is not clear how many practical implications this has. One question that would ideally be answered is: in practical settings, to what degree does bad initialization cause bad performance specifically due to bad minima? (as opposed to, say, slow convergence or bad generalization performance).
* Weight decay: the authors penalize the size of the norm of the weights as it diverges from a constant, as opposed to when it diverges from 0 as is normally done for weight decay. They show that this works as well or better than normal weight decay in a number of settings. This seem to put into question the belief sometimes held that solutions with smaller norms will generalize better.
* Kernel theory: the authors try to reproduce some of the empirical properties predicted in the Neural Tangent Kernel paper (Jacot et al., 2018) in particular by using more realistic architectures. The results, however, do not appear very conclusive. This might be the weakest part of the paper, as it is hard to draw anything conclusive from their empirical results.
* Rank: The authors challenge the common belief that low rank provides better generalization and more robustness towards adversarial attacks. When enforcing a low or high rank weight matrices during training on ResNet-18 trained on CIFAR-10, the two settings have similar performance and are similarly robust to adversarial attacks, showing at least one counter example.
I think overall this is a useful although somewhat incremental paper, that makes progress in the understanding of the behavior of neural networks in practice, and can help guide further theoretical work and the  development of new and improved training techniques and initialization regimes for deep learning.
Other comments/notes:
* minor: the order of the last 2 sub topics covered (rank and NTK) is flipped in the introduction, compared to the abstract and the order of the chapters
* in the table confidence intervals are given, it would be nice to have more details on how they are computed, (e.g. +- 1.96 * std error)
* how is the constant \mu in the norm-bias chosen?