Hi,
I was looking at the  implementation of the AdamW optimizer 5 and I didn’t understand why you put the weight decay at the end.
Shouldn’t you swap between this line:
p.data.addcdiv_(exp_avg, denom, value=-step_size)
and the weight decay part?
Thanks.
The AdamW algorithm from the “DECOUPLED WEIGHT DECAY REGULARIZATION” paper & The relevant source code for transformers.AdamW:

image1681×794 201 KB