Mike2004:

someone explain me better, what the weight decay parameter in optimizer ADAM, does?
Thank you.


The weight_decay parameter adds a L2 penalty to the cost which can effectively lead to to smaller model weights.

  
    
    
    How does SGD weight_decay work? autograd
  
  
    Hello, 
i write a toy code to check SGD weight_decay. 
but it seems to have no effect to the gradient update. 
am i misunderstand the meaning of weight_decay? 
thank you very much. 
PyTorch 1.0 

import torch
import numpy as np

np.random.seed(123)
np.set_printoptions(8, suppress=True)

x_numpy = np.random.random((3, 4)).astype(np.double)
w_numpy = np.random.random((4, 5)).astype(np.double)
x_torch = torch.tensor(x_numpy, requires_grad=True)
w_torch = torch.tensor(w_numpy, requires_grad=True)

l…
  


https://dejanbatanjac.github.io/2019/07/02/Impact-of-WD.html 392