KickItLikeShika:

And the problem is not the LR, when I try to manually create a new optimizer with this new optimal LR, I get better results.


This approach sounds right, as I don’t think that assigning the parameters() generator would directly work. The optimizer assigns the parameters here 1 so you could either try to reproduce this method or just create a new optimizer and set the found learning rate afterwards.