Learning rate decay is a common need during model training, right?
So we don’t have this in current Pytorch optim?
As you recommend, I wonder reconstructing the optimizer with new parameters would bring in some performance overhead, although it would be very small compared to the whole training time?