Oops! Sorry for misunderstanding your point earlier. I think what you are raising ties to the difference between weight decay and L2 regularization? And that PyTorch currently implements L2 regularization but it’s still labelled as “weight decay”?