DECOUPLED WEIGHT DECAY REGULARIZATION
L 2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L 2 regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW
This review has been somewhat challenging to complete. As the authors write, this work has already been impactful and motivated a great deal of further research. The empirical evaluation is convincing and the results have been reproduced and further studied by others. A moderate amount of space in the paper (Section 3, Section 4.5) is used to refer to work motivated by the paper itself. While I do not take issue with this I believe it should be considered for the final decision (in the sense that disentangling the contributions of the authors and related work becomes tricky). With this said, I continue with my review.
Paper summary: The authors observe that L2 regularization is not effective when using the Adam optimizer. By replacing L2 regularization with decoupled weight decay the authors are able to close the generalization gap between SGD and Adam and make Adam more robust to hyperparameter settings. The empirical evaluation is comprehensive and convincing.
Detailed comments:
1) The authors emphasize the fact that L2 regularization and weight decay are not the same for different optimizers and claim that this goes against the belief of some practitioners. In my experience, most practitioners would not be surprised by this observation itself. The second observation made by the authors, that L2 regularization is not effective in Adam, is the more interesting (and perhaps surprising) observation.
2) I am not convinced of the importance of Proposition 3. In practice, adaptive methods will have a preconditioner which depends locally on the parameters. I understand the motivation from the previous paragraph but felt that the formal result added little.
3) Section 3 introduced the Bayesian filtering perspective of stochastic optimization. The authors share the observation of Aitchison, 2018 that decoupled weight decay can be recovered in this framework. My interpretation is that this observation is important _because_ of the empirical observations in this paper and does not necessarily provide theoretical support for the approach. However, the last paragraph of Section 3 seems to utilize the Bernstein-von Mises theorem to promote the idea that with large datasets the prior distribution is unimportant (and is ignored). I am not sure that I follow this argument. For example, this claim seems to be completely independent of the optimization algorithm used and moreover Propositions 1,2, and 3 are independent of the data distribution. I suspect that this confusion is due to a misunderstanding on my part and would appreciate clarification from the authors.
4) The empirical evaluation in this paper is very strong and these practical techniques have already been adopted by the community in addition to spurring novel research. The empirical observation broadly explores two directions: decoupled weight decay leads to separable hyperparameter search spaces (meaning optimization is less sensitive to hyperparameters), and decoupled weight decay gives improved generalization (and training performance). Both claims are explored throughly with strong evidence given for the improvement due to AdamW.
Overall, I find this paper to be presented well and with convincing empirical results. I feel that the theoretical justification for decoupling weight decay are a little weak, and believe that other work is moving towards better explanations then the ones presented in this paper [1,2,3]. Despite this, I believe that this paper should be accepted.
Minor comments:
- I find the notation in the paper confusing in general. x is used to denote weights, and w to denote hyperparameters (e.g. w' for L2 regularization scale and w for weight decay scale). I don't see why it wouldn't be preferable to use the more standard W for weights, x for inputs, and lambda for hparams.
- Figure 4: it is difficult to distinguish between Adam and SGDWR (especially left).
Clarity: The paper is well written and clear. I find the notation confusing in places, but is consistent throughout.
Originality: This paper presents original findings but occasionally relies on work motivated by itself to convince the reader of its importance. I do not think that this subtracts from the value of the work.
Significance: The work is clearly significant. Even without knowing that practitioners have adopted the techniques presented in this work, the paper clearly distinguishes itself with strong empirical results.