Solved by googlebot in post #2 
              
              
                Do you know about lr schedulers? https://pytorch.org/docs/master/optim.html#torch.optim.lr_scheduler.ExponentialLR 
Optimizer recreation won’t work correctly.