Solved by DerekGloudemans in post #4 
              
              
                Yeah so weight_decay  is used to add a regularization penalty to model weights over time, to keep model weights small on average and prevent overfitting. In answer to your second question, yes, weight decay will generally tend to decay the weights for both losses, unless it was implemented in a cust…