THREE MECHANISMS OF WEIGHT DECAY REGULARIZATION
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of L 2 regularization. Literal weight decay has been shown to outperform L 2 regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the inputoutput Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.
This paper discusses the effect of weight decay on the training of deep network models with and without batch normalization and when using first/second order optimization methods. 
First, it is discussed how weight decay affects the learning dynamics in networks with batch normalization when trained with SGD. The dominant generalization benefit due to weight decay comes from increasing the effective learning rate of parameters on which batch normalization is applied. The authors therefore hypothesize that a larger learning rate has a regularization effect.
Second, the role of weight decay is discussed when training with second order methods without batch normalization. Under the approximation of not differentiating the curvature matrix used in second order method, it is shown that using weight decay is equivalent to adding to the loss an L2 regularization in the metric space of the curvature matrix considered. It is then shown that if the curvature matrix is the Gauss-Newton matrix, this L2 regularization (and hence the weight decay) is equivalent to the Frobenius norm of the input-output Jacobian when the input has a spherical Gaussian distribution. Similar arguments are made about KFAC with Gauss-Newton norm. The generalization benefit due to weight decay in this case is claimed based on the recent paper by Novak et al 2018 which empirically shows a strong correlation between input-output Jacobian norm and generalization error.
Finally, the role of weight decay is discussed for second order methods when using batch normalization. In this case it is discussed for Gauss-Newton KFAC that the benefit mostly comes from the application of weight decay on the softmax layer and the effect of weight decay on other weights cancel out due to batch normalization. A comparison between Gauss-Newton KFAC and Fischer KFAC is also made. Thus the generalization benefit is presumably attributed to the second order properties of KFAC and a smaller norm of softmax layer weights.
Comments:
The paper is technically correct and proofs look good.
I have mixed comments about this paper. I find the analysis in section 4.2 and 4.3 which discuss about the role of weight decay for second order methods (with and without batch-norm) to be novel and insightful (described above). 
But on the other hand, I feel section 4.1 is more of a discussion on existing work rather than novel contribution. Most of what is said, both analytically and experimentally, is a repetition of van Laarhoven 2017, except for a few details. It would have been interesting to carefully study the effect of weight decay on the gamma parameter of batch-norm which controls the complexity of the network along with the softmax layer weights as it was left for future work in van Laarhoven 2017. But instead the authors brush it under the carpet by saying they did not find the gamma and beta parameters to have significant impact on performance, and fixed them during training.  I also find the claim of section 4.1 to be a bit mis-leading because it is claimed that weight decay applied with SGD and batch normalization only has benefits due to batch-norm dynamics, and not due to complexity control even though in Fig 2 and 4, there is a noticeable difference between training without weight decay, and training with weight decay only on last layer. Furthermore, when hypothesizing the regularization effect of large learning rate in section 4.1, a large body of literature that has studied this effect has not been cited. Examples are [1], [2], [3]. 
I have other concerns which mainly stem from lack of clarity in writing:
1. In the line right above remark 1, it is not clear what “assumption” refer to. I am guessing the distribution of the input being spherical Gaussian?
2. In remark 1, regarding the claim about the equivalence of L2 norm of theta under Gauss-Newton metric and the Frobenius norm of input-output Jacobian, why does f_theta need to be a linear function without any non-linearity? I think the linearity part is only needed for the KFAC result.
3. In remark 1, what does it mean by “Furthermore, if G is approximated by KFAC”? For linear f_theta, given lemma 1 and theorem 1, the claimed equivalence always holds true, no?
4. In the 1st line of last paragraph of page 6, what are the general conditions under which the connection between Gauss-Newton norm and Jacobian norm does not hold true?
5. In figure 5, how are the different points in the plots achieved? By varying hyper-parameters?
A minor suggestion: in theorem 1 (and lemma 1), instead of assuming network has no bias, it can be said that the L2 regularization term does not have bias terms. This is more reasonable because bias terms have no effect on complexity and so it is reasonable to not apply weight decay on bias.
Overall I think the paper is good *if* section 4.1 is sorted out and writing (especially in section 4.2) is improved. For these reasons, I am currently giving a score of 6, but I will increase it if my concerns are addressed.
[1] a bayesian perspective on generalization and stochastic gradient descent
[2] Train longer, generalize better: closing the generalization gap in large batch training of neural networks
[3] Three Factors Influencing Minima in SGD