How does AdamW weight_decay works for L2 regularization?
Hello I have some questions about weight regularization in Adam.
Apparently the `weight_decay` in the AdamW function https://huggingface.co/transformers/main_classes/optimizer_schedules.html#adamw-pytorch has the same impact as an `L2 regularization`.

My questions are: is that parameter the same as `lambda` that we have in the regularization term?

![image](https://user-images.githubusercontent.com/36989932/88071850-3f78a880-cb6c-11ea-990c-2d97620e4b5c.png)

How does it exactly work? And what is its impact on the model complexity?
Hi there. General questions like this are probably better asked on the [forum](https://discuss.huggingface.co/). There is a research category that is exactly fitted for this. The `weight_decay` does correspond to the lambda you mention, though it's applied directly to the gradient, to avoid wasting compute with this huge some of all the weights scared.

You can also look at the [AdamW paper](https://arxiv.org/abs/1711.05101) for more information. <|||||>@sgugger thank you for your answer. I will check out the paper.