Solved by beaupreda in post #2 
              
              
                Hello, 
Yes, you should still use optimizer.zero_grad() since  the only thing it does is clearing the gradients. When you load a checkpoint, you load the weights of your model (parameters), you do not load any gradients. So if you keep training from this checkpoint, you still want to reset your grad…