Hi
@julioeu99 weight decay in simple terms just reduces weights calculated with a constant(here 1e-2). This ensures that one does not have large weight values which sometimes leads to early overfilling.
Weight decay sometimes makes the model to converge slower.
By default pytorch has weight_decay=0
Some useful discussions on the same:

  
    
    
    How does SGD weight_decay work? autograd
  
  
    The part that I circled doesn’t seem right to me: 
 [39a9f417cb836820174d71e9a9318d5ccc6191f3] 
In L2 regularization, you modify the cost as follows 
 [text_Regularized] 
The weight update should be then 
 [text_updatedWeig] 
The way PyTorch applied the weight decay seems correct to me (you can drop the factor 2)
  



  
    
    
    How pytorch implement weight_decay? 
  
  
    That line means, in other notation: 
d_p = d_p + weight_decay * p.data. 
Here’s a good article about why the L2 penalty is implemented by adding weight_decay * weight_i to the gradient: https://stats.stackexchange.com/questions/29130/difference-between-neural-net-weight-decay-and-learning-rate